Simple Digital Twin in Omniverse and Unreal Engine 5 - uic-evl/DOE_DigitalTwin GitHub Wiki

This guide walks you through setting up a simple digital twin of the Elephant Robotics MyArm300 or Cobot280 using NVIDIA Omniverse and the Unreal Engine 5 on Windows 11. In this configuration, you will need:

• A Win11 computer with NVIDIA Omniverse and Isaac Sim installed. This will be the host computer.
• A Win11 computer with Unreal Engine 5.3, NVIDIA Omniverse, Isaac Sim, and the Unreal Omniverse 5.3 Connector installed. This will be the client computer.
• An Elephant Robotics arm

All machines must be on the same wifi network. If you do not wish to use multiple computers, you can ignore the remote Nucleus connection setup, and connect Isaac Sim and Unreal to the same localhost Nucleus.

Task Summary:

1. Omniverse Setup
   1. Nucleus Setup
2. Isaac Sim Scene Setup
   1. Robot Files
   2. Robot Descriptor
   3. Python Scripts
3. Arm Code
4. Unreal Engine Connector

The main engine driving this digital twin application is NVIDIA Omniverse.

Step 1: Download Omniverse

• If you don’t already have one, you will need to create an NVIDIA Developer account.

Step 2: Create Nucleus

1. Create a Local Nucleus with the button above
   • You can leave the local data path as the default
2. Create a username and password. This can be anything you want, but you will need the username and password later, so make sure to record it.

We now have a nucleus folder called "localhost". This is where we will store the USD scene file to open in Isaac Sim and Unreal Engine.

Step 3: Configure a Remote Nucleus

To connect to this nucleus from another machine, you will need to add another user.

1. Open localhost:3080 in your browser on the host computer.
2. Click the enable sharing button to allow remote connections
   • Make sure to click to allow creating new accounts
   • You will need to allow some network pop-ups.

3. Navigate to the "Exchange" tab of the Omniverse Launcher and install "Nucleus Navigator".
   1. In the Nucleus Navigator, right click on the "localhost" directory, and click on "Administration". You will not see this option if you are not using the Nucleus Navigator app installed from Exchange.
   2. In the Administration menu, click on “Users”, then “Add User”
   3. Create a new user. Here, we will name the new user “client”
   4. Click on the client in the Users menu, click "Grant Admin Access", then click "Invite User". This will copy a link to your clipboard.

5. Paste the link into a browser. On this page, set a password for the new user.

We now have a remotely accessible Omniverse Nucleus with two users, one for the host and one for the client. We will connect the client later. For now, we will move on to the Issac Sim setup on the host machine.

Isaac Sim is a robotics simulator within the Omniverse ecosystem. In Isaac Sim we will create a USD stage to hold our robot, and this USD file will communicate with the robot and connect to Unreal.

Before we can set up our stage in Issac Sim, we need to download some files. To make the remote access simpler, we are going to save all the necessary files locally on each machine. Create a folder on the C drive named "Omniverse_Files".

1. Download the Elephant Robotics ROS repository to the C:\Omniverse_Files folder.
2. Download vr_input.py and vr_ur.py from our repository and add to the Omniverse_Files folder.
3. Download the robot_data folder from our repository and add to the Omniverse_Files folder

We can now set the stage in Isaac Sim. Launch Isaac Sim from the Omniverse Launcher.

Step 1: Add Objects to Scene

1. Import robot URDF with "Isaac Utils > URDF Importer".
   • Under import, set the “Input File” to C\Omniverse_Files\mycobot_ros\mycobot_description\urdf<name_of_arm><name_of_arm>.urdf.
   • Output Directory should be "(same as source)"
   • If it is not checked, check "Create Physics Scene" in the Input Options.
   • Import the URDF.
2. Update the velocity values in the URDF file.
   1. Open the URDF you imported in a text editor.
   2. Use ctrl + f to Find and Replace velocity = "0" with velocity = "1"
3. Add a ground plane in from Create > Physics > Ground Plane
4. If there is not already a light on the stage, add one from Create > Light > Dome Light.

Step 2: Arm Articulation Root Setup This step is ONLY for Isaac Sim versions older than 4.0.

1. Expand the "firefighter" robot arm object, and click on "joint1"
2. Find the "Articulation Root" under "Physics" in the "Properties" panel, and remove it.
3. Click on "firefighter", and click "Add" under the "Properties" panel, and "Add > Physics > Articulation Root".

Step 3: Create Robot Description File

1. Open "Isaac Utils > Lula Robot Description Editor". This is where we will set up and save the robot_description.yaml file.
2. Open "Isaac Utils > Workflows > Articulation Inspector". This will help us identify the location of joints.

3. Press the play button on the far left to start the simulation, and under "Select Articulation" on the two windows, select "/World/firefighter".

4. In the Lula Robot Description Editor, in the Command Panel, set all the joints to "Active Joint".
5. Create Collision Spheres for each joint
   1. In the Selection Panel, under Select Link, start by selecting "joint1".
   2. In the Stage panel, expand "joint1". Notice that the name of the joint for this object is "joint2_to_joint1". In the Articulation Inspector, you can use the sliders to move "joint2_to_joint1" so that you can identify where it is.
   3. Set the joint back to 0 in the Articulation Inspector.
   4. Back in the Lula Robot Description Editor, go to the Link Sphere Editor.
   5. Expand the "Add Sphere", "Scale Spheres in Link", and "Clear Spheres in Link" drop-down menus.
   6. Under "Selection Panel" start with the base most joint. For arms this will either be /joint1 or /base. For quadruped or car like robots, it should be something along the lines of /main_body. A sphere at /Base is shown with the MyArm 300 pi robot.
   7. Click "Add Sphere" to your robot's respective “joint1”. This will add a sphere as seen in the figure below.
   8. The goal is to use fit these the sphere inside of the joint1 visual. the specific visual can be highlighted by selecting the joint1 xform in the stage hierarchy viewer. In a similar location, the collision_sphere object should now be there. Use that object to translate the location of the sphere as necessary. Finally use the "Scale Spheres" button under the “Scale Spheres in Link” tab to fit the sphere inside the "joint1" visual. The “Scaling Factor” of the “Scale Spheres in Link” can be set to “0.9” to incrementally shrink, or “1.1” to incrementally grow. The sphere should be as close to the rotation point of the link, and as close to the geometry size as possible (example below).

7. Repeat this process for all of the joints in the "Select Link" field. If you are using the Articulation Inspector to identify joints, make sure you reset them all to 0 before you place a sphere. If your robot is not in the default position, but the joint positions all appear as 1, click "REFRESH" under “Update controllers” with a joint selected in the Stage panel. Be careful to not move the joints too much, or you may break the physics simulation.
8. You should end up with something like this:

9. In the Lula Robot Description Editor, set the Output file under the “Export Robot Description File” to “C:\Omniverse_Files\robot_data\robot_description.yaml”
   • Double check in the file explorer that the file saved.
10. Stop the simulation

Step 4: Add Python Scripts

1. Click on “firefighter” in the Stage view, and click “Add > Python Scripting” in the Properties panel.

2. Add "vr_input.py" and "vr_ur.py"
3. Open vr_input.py in a text editor to update files paths
   1. Set the PATH_BASE variable on line 40 to C:\\Omniverse_Files
   2. Set the end effector joint name on line 85 to the name of the final joint on the robot. This will be the joint object in the scene view that doesn’t have a joint child object when expanded. Here, this is joint6_flange

3. On line 101, update the yaml file path of robot_description_path to the name we saved from the Lula Robot Description Editor, robot_description.yaml
4. On line 102, update the URDF file of urdf_path to C:\Omniverse_Files\mycobot_ros\mycobot_description\urdf<name_of_arm><name_of_arm>.urdf. This should be the same URDF file you imported.
5. Setting the IK target. * The IK target will be the object that the arm follows. This can either be the end effector, or a disconnect object. For this guide, we will use a target cube.
   1. Create a cube with "Create > Mesh > Cube". Drag the Cube up to "firefighter” to make it a child object.
   2. Scale and place the cube so that it is roughly in the size and position of the end effector. Rename this Cube "Target".

• If you are using the end effector, open vr_input.py in a text editor and change the word "Target" on line 142 to the name of the end effector object.

5. Save USD Scene in the localhost Nucleus file. Then, reload the USD file from the Content browser.
6. Press Play on the simulation, select Target, and translate it down the Z axis slightly to confirm that the IK is working.

• If the console has an "IK failed" warning, try moving the Target closer to the robot.
• If moving the Target when the simulation is running does not move the arm, and there is no "IK failed" warning, double check all paths and file names.

Next, we are going to create and join a Live Session to view our Isaac Sim scene in the Unreal Engine from a client computer. On a second computer, make sure you have Unreal Engine 5.3 and the Omniverse Unreal 5.3 Connector installed.

As with the host computer, create a folder on the C drive named "Omniverse_Files".

Note: To install the Unreal Engine 5.3 version go to the Library tab in Unreal Engine 5 and press the + button beside the word Library to add past versions. Then 5.3 version should pop up and press download.

1. Download the Elephant Robotics ROS repository to the C:\Omniverse_Files folder
   1. When the URDF is imported on the host, it creates a folder for the USD file and the materials in place from where the URDF was imported. You will need to copy this folder from the host to the client so that it has the same file path.
   2. You will also need to open the URDF file in a text editor and change the velocity values from 0 to 1.
2. Download vr_input.py and vr_ur.py from this repo and add to the Omniverse_Files folder.
3. Download the robot_data folder from this repo and add to the Omniverse_Files folder

Now, we can begin to connect to the host.

1. In the Omniverse Nucleus Navigator, click on "Connect to a Server"
2. Enter the IP address of your host computer. Log in with the credentials of the client user you created earlier.
3. Launch Unreal Engine 5.3 and create a new Blank project.
4. If you have the Omniverse Unreal Connector installed, you should see an Omniverse icon in the top left corner.

5. Click on the Omniverse Icon, then "Add Server". Enter the IP address of the host machine.
6. In the Content Browser or Content Drawer, you should see a folder called "Omniverse". In this folder, navigate to where you saved the USD on the host.
7. On the host computer, start a Live Session in Isaac Sim from the icon on the top right.
8. On the client, join the Live Session by clicking the "Omniverse icon > Live Session".
9. Now, if you start the simulation on the Host, the arms shown on both machines should move in sync.

On the arm:

1. Download link_<name_of_arm>.py from this repository in the Omniverse_ER folder.
   • The mc variable will hold the communicator for your hardware. Make sure it is being passed the correct port and baud, or the script will not work.
   • Make sure the joint_angles list has the correct number of angles for your robot (equivalent to the number of motors).
2. pip install zmq
   • If this does not work, you may need to sudo apt-get update && sudo apt-get upgrade

In Host Isaac Sim:

1. Add vr_ur.py to the "firefighter" object. If you have not already.
2. Update the IP Address on line 36 to the IP address of the arm you wish to connect to.
3. Save and reload the scene.

Now, you should be able to run link_<name_of_arm>.py on the arm, and start the simulation on the host Isaac Sim to control the physical arm.

Isaac Sim:

If you open an existing USD from an Omniverse Nucleus, Issac Sim may not be able to recognize the USD for the arm. In this case, the "firefighter" object in the Stage view will have its name written in red. To solve this, click on "firefighter" in the Stage view and go to "Properties > References > Asset Path". Use the folder icon to the right to find the USD file for the arm on your local drive.

Arm:

To find more details and examples of the ER Python API, check out their demo repository.

Port not free:

1. sudo netstat -ltnp to find the process on port 5560
2. kill -9 <pid>

To find IP Address: ifconfig

Isaac Sim linking script: python link.py

• On the MyArm300 Pi, you can use PI_PORT and PI_BAUD to get the port and baud.
• On the MyCobot280 JN, the port is /dev/ttyTHS1 and the baud is 1000000.

It can be useful to test the arm with an ROS2 node

• Must use ROS2 terminal, or source it in regular terminal with source /opt/ros/galactic/setup.bash In colcon_ws/, run ros2 launch myarm_300 slider_control.launch.py

You can mirror the arm in ROS with ros2 launch myarm_300 mycobot_follow.launch.py