• Introduction
• Controlling the arm

• URScript
• ROS

• Remote Connection

The robot arm that will be used is the UR3. UR stands for Universal_robots, it is a Danish manufacturer of smaller flexible industrial collaborative robot arms. UR has a series of different robots going from UR3 to UR10, where the UR3 is the smallest robot arm:

The robot arm is made out of 7 different parts, to start of with the base(1), this is attached to the object where the robot stands on. The shoulder(2) can rotate the entire arm 360 degrees. The upper arm(3) that is attached to the shoulder, this part rotates on a different axis than the shoulder, it can also rotate 360 degrees, but mostly will find collision with the object the arm is standing on when it tries to rotate 360 degrees. The next part is the forearm(4), this rotates on the same axis as the upper arm, which reconstructs an elbow together with the upper arm. The other 3 are wrists(5-7), these are made for precision and perfect accuracy. The robot itself can hold up to 3kg, weighs 11kg and has a reach of 50cm.

There are several ways to control the robot, first and foremost with its own control panel. The arm gets delivered with a control panel, this panel makes it able to manually control it, or to run a script on the robot. It's a device that directs the sensors and servo's inside the robot and can communicate with external software. Running scripts will be managed by the control panel. Those scripts have to be send via internet, the robot has an ip adres to send commands and scripts to from external software. A simple language made specificly for the UR robot is URScript, a selfmade programming language to control the arm. Another way to control the arm is through ROS, the Robot Operating System. It is an operating system that supports all kinds of robots and offers a lot of possibilities.

UR Script is a programming language for the UR robot series. The language is based on Python so most syntax is quite similar. Most of the UR Script usage is through the built-in functions linked in the UR Script manual. All the functions have a small description on what they should do.

There are multiple ways to use URScript on the robot, you can write a program on a pc and load it on the robot using an USB stick. You can write a program using URScript on the robot itself or you use the build in interpreter. We make use of the robots interpreter. The robot hosts a socket connection on port 30002. You can send strings to this socket that the robot will try to execute. You can in fact send entire programs using this method. We connect to the socket and controll the robot using a python script.

To controll the robot we take the following steps:

1. connect to its socket
   
2. Create URScript code to execute the desired command
   
3. Send the command using the socket connection
   

If we want to read data from the robot arm it becomes a bit more complicated. To do it we must follow the following steps:

1. Host a socket on the laptop
   
2. connect with the socket of the robot arm
   
3. Write URScript code that connects with the socket on the laptop, gets the data we want, puts it in a string and sends the string
   
4. Start a thread that waits for the data from the robot
   
5. Send the URScript program to the robot
   
6. receive the data from the robot
   
7. Close the connection the robot made with the computer.
   

An important thing to know is that the robot can not run two programs at the same time. Sending a command to the robot while it is still executing the last on will cancel the last command. It will also interrupt any program running locally on the robot.

Currently we support 8 commands supported in python using URScript:

• move_up
• move_down
• move_forward
• move_backward
• move_left
• move_right
• move_open
• move_close
  These commands are fairly simple and when you execute a command it will run only once and another command wont be accepted untill the robot is done moving.
  

All movement is done with increments of 5 centimeters, except for the gripper, the gripper fully opens or fully closes. Because the gripper is not supported by URScript over sockets by default we need to send some more data. We need to send the URScript code for controlling the gripper as well, this data is present in: /tardis-world/scripts/Gripper.script. We send this entire file and the rest of the program to control the gripper at the end.
Next to controlling the robot, these commands also return some data. Every time we move the robot, the robot arm also tells use what the target angles are for the joints to reach the desired position. When we open or close the gripper, the robot continuously sends data about how open or close the gripper is in real time.

See below for an example of moving the robot up.

def getPos():
    while True:
        if socket_open("192.168.1.128", 8080, "socket_2") == True:
            textmsg("Connection successful?")
            movej(pose_trans(get_forward_kin(), p[0, 0, -0.05, 0, 0, 0]), a=1.2, v=1.05, t=0, r=0)
            socket_send_string(to_str(get_target_joint_positions()), "socket_2")
	    break
        else:
             textmsg("Connection unsuccessful")
        end
    end
end

Lets break down the code:
socket_open() tries to connenct to the socket on the pc. While it is not connecting it shows that in the log of the robot using textmsg(). Once it has been connected it executes the move command (movej). There are some interesting parameters in the movej function, lets look at those:

• get_forward_kin() This function gets the current position of the tool in tool space.
  
• p[0, 0, -0.05, 0, 0, 0] This list indicates the desired change of the robot. the p indicates that the values in the list are coordinates and not joint angles. The list consists of the following values[pos x, pos y, pos z, rot x, rot y, rot z] . As you can see the z axis is currently the veritcal axis, this is because of the current rotation of the tool.
  
• pose_trans() transposes one positions on top of another. We combine this with get_forward_kin() and the desired change to get a new position in tool space.
  
• v The v stands for velocity, this indicates how fast the robot should move to the given target.
  
• a the a stands for acceleration, this indicates how fast the robot should accelarate to the velocity.
  

After we start the movement command we send the target joint angles over the socket connection. We get the angles using get_target_joint_positions(), we then turn the result into a string using to_string() and send it over "socket_2" using socket_send_string().

This piece of code and the rest of our code to controll the robot arm can be found in communication.py /tardis-world/scripts/communication.py.

Another way to controll the robot arm is using ROS. The Robot Operating System, or ROS, is free and open source software that defines components, interfaces and tools for building advanced robots. Most robots are made up of actuators, sensors and control systems. ROS helps developers to quickly build these components and then easily connect them using ROS tools. ROS will generate messages that can be easily visualized, those files are called BAG files. This makes testing much easier.

In our project we want to link the UR3 Robot to the VR headset. Since we use MQTT for linking external data the to headset, we want to use a format that supports the MQTT terms and preferably a format that we are familiar with. Luckily ROS has a lot of extensions to work with, Python is one the implemented coding languages we can work with. Therefore we will use rospy link to rospy. Rospy is a pure Python client library for ROS, the rospy client API enables Python programmers to quickly interface with ROS. The design of rospy is based around the implementation speed by the programmer. This way algorithms can be quickly prototyped and tested within ROS. It is an ideal library for non-critical path code, such as configuration and initialization code. In order to make your ROS work with python you need to initialize two parts.

1. Configure the PYTHONPATH, this is needed to make use of Python. If you use other ROS Packages in the code, you need to dynamically load their libraries into your path, this way you can import the packages.

2. Initializing your ROS Node, a node is a process that performs computation. Nodes are combined together into a graph and communicate with one another. These nodes are meant to operate at a fine-grained scale, a robot control system will usually consist of many nodes. You can call a node a script of computation.

add how to get it to work/what we are stuck on, we need to explain this further

Ros stand for Robot Operating System, though Ros is not a real operating system like Linux or Windows. It is more like an environment that makes it able to connect certain hardware with each other through software. The environment is based around having nodes, a node is a certain piece of software that can be written in C++ and Python. Each node can take care of a small subset of tasks, for example reading a sensor or controlling a servomotor. These are different tasks that in most cases want to be combined to make complete software and hardware systems. In order to be able to connect nodes with each other ROS uses a Publish-Subscribe Protocol. This means that any node can publish information in the environment, where other nodes in the network can subscribe to that certain published information. The shared information is done by making a topic and publishing it. Subscribers in a topic-based system will receive all messages published to the topics to which they subscribe. The publisher is responsible for defining the topics to which subscribers can subscribe.

To make a ros environment you need to be on a Linux operating system such as ubuntu. To install ros there are some steps you want to follow from their own website: http://wiki.ros.org/noetic/Installation/Ubuntu . After the installation you can start by setting up to environment. First off you need to make a catkin_ws folder. This folder is designed for development. With the command catkin_make inside the src folder it will start to build a development environment. In every terminal you use for executing or reading nodes you need to type source devel/setup.bash. Normally you would need to run this command each time you open up a new terminal. In order to avoid this you can go into the bashrc file with sudo nano ~/.bashrc, at the end of this file you can put the source ~/catkin_ws/devel/setup.bash so each time the script does it for you.

Catkin has the following dependencies :

• CMake
• Python
• GTest
• GNU C++ Compiler (g++)

Once the catkin_ws folder is setup correctly you can install packages, these are nodes made by others that you can use in your own environment. Most packages are on Github. In order to use a package you might need to install dependencies dependent on if the package needs them, such as a i2c library. If dependencies are needed most packages will have the information on their Github page. To use the package you need to clone the repository inside the ~/catkin_ws/src folder with ~/catkin_ws/src:$ git clone *repo*. When the package is cloned you go back to the catkin_ws folder with cd .. and use the command: catkin_make. This way catkin builds and compiles the packages ready to be launched. The next step is to launch the node, there is a folder called launch inside the package, within the folder there are multiple .launch files, A simple launch file will be displayed here:

It starts with the <node .. > and ends with , this part creates a node with the name "rviz" and the type is the file we want to run. Before we go further it is important to get used to the ros commands, otherwise it can become hard to understand commands and navigate through them: "http://wiki.ros.org/ROS/Tutorials/NavigatingTheFilesystem".

To run a node with the package, you need to open a new terminal and be inside the path of the package folder. Run the command: rosrun *the name of the package* *name of the node* These names you can find in the launch file explaned above, now the node is running and it can publish topics. In order to see topics inside the environment you need to open a new terminal and use the command: rostopic list. This will show all the current active topics in your network. If you want information about a certain topic you can type $ rostopic info *name of the topic*, it will give information about the type of the message being send out and its name. If you want to publish your topic, you type: rostopic pub /*name of the topic*. It will give outputs dependent on the package that you’re running.

To make your own package it simply starts with creating a workspace. Inside the ~/catkin/src folder you use the command: catkin_create_pkg *name of package* *depency* . catkin_create_pkg generates a package with the given name, the depencies are libraries that you want to add, such as rospy. So it can look something like this: catkin_create_pkg robot_arm rospy. In this folder it is possible to develop code with Python. There are some things that are good to know once you’ve made a package, there are certain files that are generated by the catkin workspace; the package.xml and the CMakeLists.txt. These files are used for describing your ros and what to do with the package and the description of the package etc. The package.xml is a normal xml file starts with inside the package there is information about the package and the build dependencies:

the CMakeLists.txt contains information about what to include during the build process:

Here you can see that it will try to find all the packages in order for this node to run its scripts. If you want to you run your Python file, all you need to do is add the name of the python file into the *name of the package*.launch, in here you can find names of files that need to be launched as well for the package to successfully launch. It works the same as a .xml file where you can . Some commands are important to know such as <include file=”$(find *other needed package*)/launch/*name of that launch file*”. This way you can include another launch file into the node. To launch the Python script or any script you will be needing this line inside you launch file: <node pkg=”*package name*” name=”*name of the node*” type=”*Python file*” output=”screen” ></node>. Now it will run the Python script with the following commands in a terminal: ~/catkin_ws/src/*pkg*/launch$ roslaunch *pkg* *launchfile*.launch

In our project the focus was on making the UR3 Robot arm communicate with the VR game. For us to make that possible we first need be able to make a package out of the UR3. Luckily Github has many examples of a UR3 Robot arm package that is made with rospy. Most packages have to option to test the UR3 virtually, which is a great tool to test movement and data, this way you can test commands and scripts safely without damaging the actual robot arm. To visualize the robot arm with ROS the packages use the MoveIt software. This is a open-source robotic platform. It provides motion planning, collision detection capabilities to execute difficult actions. An easy way to use the MoveIt software, is to use it with Rviz, a graphic interface program to visualize 3D models. This provides an easy-to-use motion planning interface to work with:

The idea was to setup an environment with ROS to control the entire robot with MoveIt data. This way we could have had real-time data with more possibilities for the robot, such as acceleration better data positions/angles and accurate actions. At last we could handle the camera that is on the UR3 Robot arm, for more interesting features in the VR game. When doing research about this software and getting known with ROS we had several problems with installing ROS. These were problems with setting up an actual environment following the guide on the official ROS website, the bash scripts were not working properly on the Ubuntu version that was used to test installing ROS. After trying to reinstall ROS several times, the pathways towards the file in the bash scripts would be wrong. Since sudo apt-get remove ros-* did not work for every file. After investigating these issues it ended up installing correctly, where I could immediately start making an environment with this package: https://github.com/UniversalRobots/Universal_Robots_ROS_Driver , a very well explained tutorial to test the robot arm, but in a virtual way. At first it I just wanted to test if it worked. Once the test was setup with an environment I ran a test_move.py with the visuals of Rviz:

At first the commands lead to opening the visual 3D models as normal, the model was correct and the joint positions were correct according to the control panel on the UR3 Robot. Once I started the test_move.py file, the 3D model seemed to go weird places where the robot arm would then bug out. When I tried to use the Rviz software manually with ROS and I would plan movement, the execution would send the robot the other way around, colliding it into the ground. This problem we also faced with the selfmade A-Frame virtual robot arm, it is because the angles from the robot are negative values on certain joints. This causes the robot to go to the designed target position but the other way around. So after starting to find out more about the rospy and the scripts that get send to the visuals, we had little time left to finish this project. This is where we stopped looking for possibilities with ros and started going back to the URScript from the UR3 itself.

To be able to remotely controll the robot we added an mqtt connection to the python code. This code waits for messages on the "hbo_ict_robot_arm_controll" topic. The mqtt broker we use is broker.emqx.io. This is a public mqtt broker that supports web sockets and tcp sockets over both unsecure and secure connections. We use the port: 1883 for a non secure tcp connection.

The script then executes the command given in the received message if the command is known. To achieve this we create a second python thread, to be able to handle the receiving of commands and using them in parallel. The commands have the following structure:

{
    command: "move_up"
}

Once a message has been received, the command is extracted and we check if a command is already being executed using a mutex. If the mutex is free we send the command to the movement queue.
The second thread waits for an item to be added to the movement queue and once that happens it executes the corresponding function to let the robot move in the desired way.