Introduction tutorial - REHYB/LegoArmExoskeleton GitHub Wiki

In this tutorial, a simple Unity scene will be implemented which is communicating with a LEGO EV3 motor which is powered by a Raspberry Pi + BrickPi combo through UDP. The communication is bidirectional, meaning Unity is sending messages to the motor and the motor can also send packages to Unity.

All the files and programs created during this tutorial can be found in the example project folder!

Step 1 - Hardware setup and prerequisites

You need to have Unity downloaded on your computer. This tutorial uses Unity version 2019.2.8f1.

For this tutorial, we will use the BrickPi + Raspberry Pi (RPI) control unit, an EV3 motor connected to Port B and a power bank to power up the system. (Make sure that the power supply can provide 9V, so the motor is able to operate. The XT-16000QC3 PowerBank is recommended). You can use either a Medium EV3 motor or a Large EV3 motor, it does not matter for the sake of the tutorial. It's recommended to attach some LEGO pieces to the rotating part of the motor so you can see the rotation and rotate yourself as well.

See the picture below of the hardware setup:

Furthermore you need a router which provides a closed wireless network. You have to be able to give static IP addresses to the Raspberry Pi and the computer which runs the Unity environment. How to assign static IP addresses differs between different types of routers, so check your model. In this tutorial, the Raspberry Pi has the IP address of 192.168.0.200 and the computer which runs Unity has 192.168.0.101 .

You also need to be able to control the Raspberry Pi from your computer to make things easier. The easiest way to do that is by using SSH and to access and edit the files of the Raspberry Pi to set up a Samba share.

Step 2 - Raspberry Pi setup

Make sure that the Raspberry Pi is connecting to the wireless network before loading anything up. To do so follow the steps:

  1. SSH into the RPI
  2. Run sudo raspi-config
  3. Select 3 Boot Options
  4. Select B2 Wait for Network at Boot and select Yes
  5. Save and restart the RPI

Step 3 - Raspberry Pi programming

This system has two parts:

  • A program is running on the Raspberry Pi, taking commands from the Unity program and able to send messages to Unity
  • The Unity program which is able to send data to the Raspberry Pi, therefore also actuating the motors and listening to incoming data from the Raspberry Pi

In this step we will implement the first part, the side of the Raspberry Pi. On how to develop with BrickPi by using python please visit their official documentation/examples! This tutorial will also take codes from examples which can be found there.

  1. SSH into the RPI
  2. Create a python script called unity_motor.py
  3. Add the following lines:
#!/usr/bin/env python
from __future__ import print_function # use python 3 syntax but make it compatible with python 2
from __future__ import division

import brickpi3 # import the BrickPi3 drivers
import socket # import the socket library to communicate using UDP protocol
import threading # using threading to listen to UDP messages
import time

The lines except the last three are needed for the BrickPi3 system. The socket library lets us use the UDP protocol and bind to a socket. The threading library will be used to start a thread to listen to incoming messages. The time library will allow us to slow the running of the main loop down a little bit, so the Raspberry PI CPU is not overloaded.

  1. Add the following lines below (explained in the comments):
BP = brickpi3.BrickPi3() # Create an instance of the BrickPi3 class. BP will be the BrickPi3 object.
BP.set_led(0) # For indicating succesful setup, we turn off the LED here at the beginning and turn it on if the system is succesfully set up
  1. Add the following lines below (explained in the comments):
print("Setting up UDP connection...")

# Unity program
SERVER_IP = '192.168.0.101'
SERVER_PORT = 5013

# Raspberry Pi
CLIENT_IP = '192.168.0.200'
CLIENT_PORT = 5011

# Create the socket
sock = socket.socket(socket.AF_INET, # Internet - given, don't change it
                     socket.SOCK_DGRAM) # UDP - given, don't change it

# Bind to the socket
sock.bind((CLIENT_IP, CLIENT_PORT))

print("UDP connection set up!")

The ports (and also the IP addresses as described earlier) can be changed but it is not necessary. Keep them like this for the tutorial.

  1. Add the function which will be executed once the right UDP message arrives (explained in the comments):
def moveMotor():
    target = BP.get_motor_encoder(BP.PORT_B) + 90 # Read the position of the motor and add 90 degrees, so in the next line it will be moved by 90 degrees
    BP.set_motor_position(BP.PORT_B, target)
    time.sleep(0.5) # Wait for the movement to be over and then set the motor to float again, so it can be rotated by hand.
    BP.set_motor_power(BP.PORT_B, BP.MOTOR_FLOAT)
  1. Add the message callback function and the receive message functions which will run on its own thread later. The messageCallback function gets the data as input, decodes it with standard UTF-8 and then checks if it says "move". Only in that case will move the motors. By doing that you can distinguish between different messages.
def messageCallback(data):
    message = data.decode("utf-8")
    print(message)

    if message == 'move':
        moveMotor()
    
def receiveMsg():
    while True:
        data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
        if (addr[0] == SERVER_IP):
            messageCallback(data)
  1. Add the main function and its main loop to the program:
def main():
    try:
        try:
            BP.offset_motor_encoder(BP.PORT_B, BP.get_motor_encoder(BP.PORT_B))
        except IOError as error:
            print(error)

        # Starting a thread which is listening to the UDP messages until the program is closed
        receiveMsgThread = threading.Thread(target=receiveMsg, args=())
        receiveMsgThread.daemon = True # Making the Thread daemon, so it stops when the main program has quit
        receiveMsgThread.start()

        BP.set_led(100) # Light up the LED to show the setup is successful
        print("Main loop running...")
    
        while True:       
            value_motor = BP.get_motor_encoder(BP.PORT_B) # get the position of the motor
            value_motor = str(value_motor) # turn it into a string, so it can be sent through UDP

            udp_message = str.encode(f"{value_motor}")
            sock.sendto(udp_message, (SERVER_IP, SERVER_PORT))
            time.sleep(0.01) # Without sleep the system logs and sends data to the server ~820 times per second (820 Hz)

    except KeyboardInterrupt: # except the program gets interrupted by Ctrl+C on the keyboard.
        sock.close() # close the UDP socket
        print('\nsocket closed')
        BP.reset_all()        # Unconfigure the sensors, disable the motors, and restore the LED to the control of the BrickPi3 firmware.
        print('program stopped')

if __name__ == "__main__":
    main()

The BrickPi system works in a similar fashion to the Arduino. It runs the main, never-ending loop (while True:). In this main loop the position of the motor is constantly read and being sent to the Unity program, which moves the cube accordingly. When the program has quit (by pressing Ctrl+C the socket is closed and the sensors are unconfigured. Much of this code is from the previously mentioned examples from the BrickPi3 examples.

That's it for the Raspberry Pi script. What this easy script does is:

  • Binds the IP address of the RPI to a UDP socket
  • Listens to UDP messages
  • If the message says 'move' it moves the motor by 90 degrees
  • It constantly sends the motor rotation data to Unity (we have to implement the Unity part, so it receives the message and moves the cube)

Step 4 - Unity programming

So the Raspbery Pi script is ready, now let's create a Unity program which moves a Cube based on the status of the motor and moves the motor if a key is pressed on the keyboard. All this will be done by sending UDP packages between Unity and the Raspberry Pi.

This tutorial assumes that the reader has some basic Unity knowledge. If you need refreshment or more knowledge on Unity, for great tutorials visit the official Unity learn site or watch the amazing introduction video series by Brackeys where he explains most of the concepts needed for this tutorial with just 10 short videos.

1.

First, create a Unity 3D project (For this tutorial version 2019.2.11f1 was used, however it should work with any new version of Unity)

2.

Go to the Github folder of the UdpUnityPackage (developed at the Technical University of Denmark) and download it. You can either just download it using the "Download" button, or clone it using the SSH link https://github.com/nilsrasa/UdpUnityPackage.git. See how to do it with Github

3.

Go to Assets->Import package->Custom package

4.

Search for the previously downloaded UdpUnityPackage. In the the Release folder select the UdpUnityPackage.unitypackage file and open it.

5.

In the import window leave everything selected and hit "Import"

The most important file which you have to care about is the UdpHost.cs in Udp/Scripts. Since it has its own namespace you will be able to use it anywhere in your Unity project by adding using Udp; to your C# scripts.

6.

Add an empty GameObject to you scene and name it UdpConnection.

7.

Add the previously mentioned UdpHost.cs script to the UdpConnection GameObject either by just drag-and-drop or "Add Component" -> Search for "Udp" and it should come up.

Once you have done that you should see something like this:

Let's go over the fields:

  • The "Host settings" and "Client settings" are the networking data of the Raspberry Pi and the Unity program.
  • The "Auto Connect" field if checked (True) it tries to connect to the Raspberry Pi automatically. If it's unchecked you have to call the connection at some point in the code manually.
  • In the Stream "Message" field you will be able to see the incoming messages from the Raspberry Pi in real time.

8.

Change the fields according to the network data in the Raspberry Pi section

  • Host IP and port to 192.168.0.101 and 5013 respectively
  • Client IP and port to 192.168.0.200 and 5011 respectively
  • Enable Auto Connect

Your Udp Host script in the inspector should look like this now:

9.

Let's start by making the connected LEGO motor to move whenever the key "m" (for move) is pressed on the Keyboard (like an input in a video game). To do that we will create an event system. On how to create event systems watch this great video.

  1. Create a GameObject and name it ExampleEventSystem
  2. Create a new script called ExampleEvents and assign it to the GameObject
  3. Add the line using System; to the top and copy the code overriding the Start and Update methods, so your code looks like this in the ExampleEvents script
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class ExampleEvents : MonoBehaviour
{
    // The singleton instance
    public static ExampleEvents current;

    // Singleton operation
    private void Awake()
    {
        current = this;
    }

    // The event to which we will chain the function which is sending the UDP message
    public event Action onMKeyHit;

    public void MKeyHit()
    {
        onMKeyHit?.Invoke();
    }
}

10.

Make a "GameController" which is triggering this event when the button is pressed by following these steps: Create a new GameObject, name it "Controller" and assign a new script to it named "ExampleController".

In this script we will do most of the work. We want to check when the key "m" is pressed and when it's pressed trigger the event specified previously in ExampleEvents.cs. We want to send the UDP message of "move" when the "m" key is pressed, so we write our small function which we add to the subscribed events.

Later this controller will also be responsible for listening to the UDP messages and move the cube as described in the following sections, but for now, let's only focus on sending the "move" command FROM Unity.

To use UDP, we add using Udp; to the top. Then in the class we create an instance of the UdpHost.

public UdpHost udpHost;

We create a function which will be called whenever the key "m" is pressed. In this function we are using the previously created udpHost instance and send a string message "move"

private void OnMKeyhit()
    {
        udpHost.SendMessage("move");
    }

In the Start method we add the function to the list of event subscribers:

void Start()
    {
        ExampleEvents.current.onMKeyHit += OnMKeyhit;
    }

In the Update method we are checking whether the key "m" is pressed and if it is we are triggering the event. Use GetKeyDown, so it's only registered once, when the key is pressed.

void Update()
    {
        // Checking when the key "m" is pressed
        if (Input.GetKeyDown("m")) {
            // If it's pressed, trigger the event which we specified in our ExampleEvents.cs
            ExampleEvents.current.MKeyHit();
        }
    }

After adding all these building blocks, our ExampleController.cs should look like this:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Udp;

public class ExampleController : MonoBehaviour
{
    public UdpHost udpHost;

    // Start is called before the first frame update
    void Start()
    {
        ExampleEvents.current.onMKeyHit += OnMKeyhit;
    }

    // Update is called once per frame
    void Update()
    {
        // Checking when the key "m" is pressed
        if (Input.GetKeyDown("m")) {
            // If it's pressed, trigger the event which we specified in our ExampleEvents.cs
            ExampleEvents.current.MKeyHit();
        }
    }

    private void OnMKeyhit()
    {
        udpHost.SendMsg("move");
    }
}

Going back to the inspector, you should drag-and-drop the UdpConnection gameobject to the freshly created slot in the Example Controller, so it should look like this:

11.

Let's create a cube which will move horizontally now based on the values from the LEGO motor which are being streamed through UDP. So add a 3D GameObject Cube to your scene in Unity.

12.

Now let's expand the ExampleController.cs so it listens to the UDP messages arriving from the Raspbery Pi and moves the cube horizontally.

  1. In your ExampleController.cs script add a public Transform variable which represents the cube. We will move it by changing this Transform to move the cube.
public Transform cube;
  1. Add a private float variable which we will use for changing the position. It has to be another variable, because it is not possible to change Transform from not the main thread (where the UDP message listening is happening).
private float motor_angle;
  1. Create the function which will be called every time a new message arrives from the Raspberry Pi. In this case we are just setting the previously created variable to a new value, but one can do anything here, for example printing out the incoming value and so. Keep in mind, that whatever is executed in this function is executed in another thread than the main, so some functionalities (e.g. changing Transform) will be limited.
private void OnMotorValue(string value)
    {
        motor_angle = float.Parse(value);
        //Debug.Log("This is the received value: " + value);
    }
  1. Add this function to the subscribers of the UdpHost's OnReceiveMsg event to the Start method.
UdpHost.OnReceiveMsg += OnMotorValue;
  1. Finally let's move the cube based on the previously registered value (motor_angle). We do this in the Update method. The value is divided by a 100, so the movement is slower and smoother.
cube.position = new Vector3(motor_angle / 100f, 0f, 0f);

After all these changes our extended ExampleController.cs should look like this:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Udp;

public class ExampleController : MonoBehaviour
{
    public UdpHost udpHost;
    public Transform cube;
    private float motor_angle;

    // Start is called before the first frame update
    void Start()
    {
        ExampleEvents.current.onMKeyHit += OnMKeyhit;
        UdpHost.OnReceiveMsg += OnMotorValue;
    }

    // Update is called once per frame
    void Update()
    {
        // Checking when the key "m" is pressed
        if (Input.GetKeyDown("m")) {
            // If it's pressed, trigger the event which we specified in our ExampleEvents.cs
            ExampleEvents.current.MKeyHit();
        }

        cube.position = new Vector3(motor_angle / 100f, 0f, 0f);
    }

    private void OnMKeyhit()
    {
        Debug.Log("pressed");
        udpHost.SendMsg("move");

    }

    private void OnMotorValue(string value)
    {
        motor_angle = float.Parse(value);
        //Debug.Log("This is the received value: " + value);
    }
}

13.

Drag and drop the Cube to the new field of the Example Controller and the system is ready to run

Step 5 - Running the system

The start order doesn't matter since the Unity UDP package is constantly trying to connect. So you can start the Python program first on the Raspberry Pi, or the Unity program on the PC. This makes the system robust.

To start the Python program:

  1. SSH into the RPI
  2. Go to the folder where the script is located
  3. Type python3 unity_motor.py

You should see the messages

Setting up UDP connection...
UDP connection set up!
Main loop running...

And every time the "move" message arrives another text saying printing move.

Start the Unity program as well and the system should work. Rotate the LEGO motor of the Raspberry Pi to see the cube moving horizontally in Unity. While selecting the Game view in Unity press the key "m" and see the LEGO motor moving 90 degrees.

Congratulations, you have built you first program using Unity and the LEGO motor system with BrickPi and UDP.

Optional: Run the RPI script on startup

If you wish to have the LEGO program start automatically once it's powered up, please follow the steps below. This is beneficial, because then you don't have to SSH into the Pi and manually start the program

  1. Boot up the Raspberry Pi
  2. SSH into the Pi (https://www.raspberrypi.org/documentation/remote-access/ssh/) (By default it's: ssh [email protected])
  3. Go to sudo raspi-config and set in the boot options to wait for network (this has been explained before, just make sure it's set, otherwise the script will run into an error because the network hasn't been found yet).
  4. Open up the profile file sudo nano /etc/profile
  5. At the end place the following: /usr/bin/python3 <PATH_TO_SCRIPT> and save it
  6. On the next bootup of the Pi - if there's network connection (which is required for the communication) - the program will start automatically. Once the LED on the BrickPi cease to blink and instead just brightly light, the system is up and running and connected to the UDP socket.