Overview

A human and a robot log in to the same virtual space. The human can give a cleanup instruction to the robot by pointing.

The human logs in to the virtual space using Windows + VR Headset.
The robot logs in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System).

Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space.

This tutorial can be run on a single Windows PC. However, multiple Unity editors cannot be started, so the application must be built.

Prerequisites

Common prerequisites

This tutorial uses port number 7777 for network communication.

• Make sure that anti-virus software does not block communication.
• The port number can be changed.

Prerequisites for Human (VR)

Windows environment

Basic requirements for using SIGVerse assets are needed.

VR Headset

You need to meet the system requirements for the VR headset to work.
It has been tested with Meta Quest 2.

Please install Oculus Software to use Meta Headsets. Meta Oculus Link is also required.

Prerequisites for Robot (ROS)

• Windows environment and Ubuntu environment
  See here for details.

Environment Construction

Set the IP address and port number of the server

Open the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene.
Configure the settings in the Unity Transport script attached to the NetworkManager object.
Set the IP address of the server to [Address]. Set 127.0.0.1 if checking on a local PC.
The default port number is 7777.

Execution Procedure

If you want to run as a human (VR), please follow here.
If you want to run as a robot (ROS), please follow here.

However, the host must log in first. The host is a client that includes server functions.
The host can be switched on the login window.

Human (VR) Execution procedure

Startup Procedure

1. Launch the VR software(e.g. Oculus software).
2. Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to here.
3. Click the [Human] button to log in as a human.
4. If the robot is not logged in, wait for the robot to log in.

Windows (Login Screen)

Run

Operate the human avatar and give cleanup instructions to the robot.

The basic flow is as follows.

1. Send the message "Pick it up!" to the robot while pointing at the target object.
2. Send the message "Clean up!" to the robot while pointing at the destination.

The operation of the VR controller is as follows.

• When pull the trigger, the avatar's hand will be in the pointing position.
• When the front button is pressed, the message "Pick it up!" is sent to the robot. (A or X button for Touch Controller)
• When the back button is pressed, the message "Clean up!" is sent to the robot. (B or Y button for Touch Controller)
• When you tilt the left stick, the avatar moves in the tilted direction.
• When the right stick is pressed, the message "Good!" is sent to the robot.
• When the left stick is pressed, the message "Bad!" is sent to the robot.

Windows (Details differ from the latest version)

Reference Video (Details differ from the latest version)

Robot (ROS) Execution procedure

Startup Procedure

Start the Ubuntu side and then the Windows side.

Ubuntu side startup procedure

Open a new terminal and run the following command:

roslaunch sigverse_hsr_teleop_key teleop_key.launch

To start rviz, execute the following command.

roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch

Windows startup procedure

1. Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to here.
2. Click the [Robot] button to log in as a robot.
3. If a human is not logged in, wait for the human to log in.

Windows (Login Screen)

Run

You can control HSR by keyboard operation on the teleop_key.launch terminal.

Operate the robot and grasp one of the objects on the desk, then bring it to the place where the human points.

• Check the terminal for details of the operation.

If you want to finish, stop the Unity side and then the ROS side.

Ubuntu side (Details differ from the latest version)

Windows side (Logged in) (Details differ from the latest version)

---

Reference Video (Details differ from the latest version)