vrx_2019 results - osrf/vrx GitHub Wiki

[TOC]

VRX 2019 Results

Phase 3: VRX Challenge

VRX 2019 Results Announcement

At RobotX Interactive in Singapore we announced the results of the VRX Challenge and showed a number of highlight videos. That presentation is available here.

The Final Standings are listed on the RobotX website.

Overall Ranking

The following table includes the overall VRX Challenge ranking for each team. The "Total" is the sum of the rank scores for each task.

Team Affiliation Total Ranking
NIROM Korea Advanced Institute of Science and Technology (KAIST) 12.5 1
gt-marine-robotics Georgia Institute of Technology 14 2
usyd_robotx University of Sydney 19 3
bumblebee National University of Singapore 20.5 4
uf University of Florida's Machine Intelligence Laboratory 25 5
tang National Taiwan University 42 6
australis2 Flinders University 45 6
McGill McGill University 48 6
owltonomous Florida Atlantic University 51 6
Team_Kanaloa University of Hawaii at Manoa 53 6
Erau_Minion* Embry-Riddle Aeronautical University -- 6

Ranking by Task

The following table includes the rank score for each of the six VRX tasks. The rank score is for each task is derived from the task score as described in the VRX Competition and Task Descriptions document.

Team Station keeping Wayfinding Perception Nav challenge Dock Scan and dock
bumblebee 5 3 3 5 1.5 3
uf 1 6 4 4 5 5
usyd_robotx 3 4 5 2 3 2
australis2 6 7 7 8.5 8 8.5
gt-marine-robotics 4 2 2 1 4 1
NIROM 2 1 1 3 1.5 4
tang 8 8 6 6 8 6
McGill 9 5 9 8.5 8 8.5
owltonomous 7 10 9 8.5 8 8.5
Team_Kanaloa 10 9 9 8.5 8 8.5
Erau_Minion* -- -- -- -- -- --

Scoring by Task

The table below reports the overall score for each team on each of the six VRX 2019 tasks. Each task score represents the summary of task performance over six trial runs for each task. The details of how each task is scored are available in the VRX Competition and Task Descriptions document.

Team Station keeping Wayfinding Perception Nav challenge Dock Scan and dock
bumblebee 0.90 0.41 2.67 920.61 60 80
uf 0.11 16.20 2.72 798.22 30 30
usyd_robotx 0.19 5.14 6.47 591.59 50 100
australis2 1.01 34.28 8.42 1080 0 0
gt-marine-robotics 0.44 0.10 2.48 319.91 40 140
NIROM 0.16 0 0.78 684.76 60 70
tang 5.15 46.47 7.95 983.36 0 20
McGill 9.38 11.51 10 1080 0 0
owltonomous 2.11 95.46 10 1080 0 0
Team_Kanaloa 13.32 95.20 10 1080 0 0
Erau_Minion* 17.97 65.88 -- -- -- --

Individual Task Scores

The scores for each of six runs for each of the six tasks are reported below.

Station keeping

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
bumblebee 0.78 0.59 1.45 0.79 0.59 1.20
uf 0.02 0.04 0.35 0.04 0.10 0.11
usyd_robotx 0.08 0.08 0.57 0.02 0.22 0.15
australis2 0.03 0.10 5.63 0.05 0.15 0.09
gt-marine-robotics 0.67 0.37 0.82 0.04 0.33 0.39
NIROM 0.01 0.01 0.87 0.01 0.01 0.03
tang 6.16 1.22 6.69 6.11 5.57 5.16
McGill 3.25 10.19 9.64 11.61 11.30 10.28
owltonomous 1.89 2.80 3.46 0.83 2.74 0.96
Team_Kanaloa 7.77 34.02 13.03 3.46 8.12 13.52
Erau_Minion* 10.58 25.36 -- -- -- --

Wayfinding

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
bumblebee 0.70 0.32 0.30 0.63 0.29 0.21
uf 0.22 5.95 31.47 0.22 0.09 59.25
usyd_robotx 0.04 23.73 6.92 0.04 0.05 0.04
australis2 0.31 13.54 134.39 0.03 8.57 48.82
gt-marine-robotics 0.06 0.14 0.08 0.03 0.16 0.15
NIROM 0 0 0.01 0 0 0
tang 10.73 31.49 168.34 13.82 17.16 37.28
McGill 0.18 17.24 27.93 0.82 0.54 22.32
owltonomous 39.50 73.12 172.01 43.71 99.82 144.58
Team_Kanaloa 39.30 73.17 171.03 43.57 99.79 144.35
Erau_Minion* 34.17 97.60 -- -- -- --

Perception

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
bumblebee 0.15 2.64 4.45 0.14 2.61 6.05
uf 3.39 1.33 4.78 0.06 2.55 4.19
usyd_robotx 4.67 5.00 6.71 7.33 7.00 8.12
australis2 10.00 6.26 8.83 6.68 8.75 10.00
gt-marine-robotics 0.07 1.36 5.93 0.07 2.55 4.91
NIROM 0.14 0.14 2.01 0.01 0.15 2.12
tang 10.00 8.00 7.65 7.33 8.00 6.71
McGill 10.00 10.00 10.00 10.00 10.00 10.00
owltonomous 10.00 10.00 10.00 10.00 10.00 10.00
Team_Kanaloa 10.00 10.00 10.00 10.00 10.00 10.00
Erau_Minion* -- -- -- -- -- --

Navigation challenge

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
bumblebee 55.52 166.67 240.00 51.75 166.67 240.00
uf 53.97 51.17 240.00 46.41 166.67 240.00
usyd_robotx 23.99 31.46 240.00 24.10 32.04 240.00
australis2 133.33 166.67 240.00 133.33 166.67 240.00
gt-marine-robotics 14.26 15.57 22.33 13.02 14.73 240.00
NIROM 33.50 166.67 59.22 18.70 166.67 240.00
tang 133.33 166.67 240.00 36.69 166.67 240.00
McGill 133.33 166.67 240.00 133.33 166.67 240.00
owltonomous 133.33 166.67 240.00 133.33 166.67 240.00
Team_Kanaloa 133.33 166.67 240.00 133.33 166.67 240.00
Erau_Minion* -- -- -- -- -- --

Dock

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
bumblebee 0 0 20 20 20 0
uf 0 20 0 10 0 0
usyd_robotx 0 20 10 20 0 0
australis2 0 0 0 0 0 0
gt-marine-robotics 20 0 20 0 0 0
NIROM 20 20 0 20 0 0
tang 0 0 0 0 0 0
McGill 0 0 0 0 0 0
owltonomous 0 0 0 0 0 0
Team_Kanaloa 0 0 0 0 0 0
Erau_Minion* -- -- -- -- -- --

Scan and dock

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
bumblebee 30 10 0 20 10 10
uf 10 10 0 0 0 10
usyd_robotx 30 20 10 30 10 0
australis2 0 0 0 0 0 0
gt-marine-robotics 30 30 30 10 30 10
NIROM 30 0 30 10 0 0
tang 10 0 0 0 0 10
McGill 0 0 0 0 0 0
owltonomous 0 0 0 0 0 0
Team_Kanaloa 0 0 0 0 0 0
Erau_Minion* -- -- -- -- -- --

(*) Team Erau_Minion has been evaluated for scoring purposes but not for ranking purposes.

Logs

All logs are available. See this tutorial for instructions.

Summary of submission configurations

The following table summarizes the propulsion and sensor configurations used by the VRX teams.

Team Thrusters Cameras Lidars Lidar 1 Lidar 2 P3D
mcgill_robotics 4 3 1 16_beam NA N
bumblebee 4 3 2 32_beam 32_beam N
gt-marine-robotics 4 1 1 32_beam NA N
NIROM 4 3 2 32_beam 32_beam N
Team_Kanaloa 4 2 1 16_beam NA N
owltonomous 2 2 1 32_beam NA N
australis2 3 2 1 32_beam NA N
Minion 2 3 2 32_beam 32_beam N
usyd_robotx 3 3 1 32_beam NA N
uf 4 3 1 32_beam NA N
tang 3 3 1 16_beam NA N

Phase 2: Dress Rehearsal

Lessons Learned

Phase 2 Retrospective and Lessons Learned

Logs

All logs are available. See this tutorial for instructions.

Scoring Summary

Team Station keeping Wayfinding Perception Nav challenge Dock Scan and dock
mcgill_robotics 86.96 79.87 0 0 0 0
bumblebee 7.92 14.42 108.78 0 0 0
gt-marine-robotics 9.44 79.79 89.36 571.71 0 10.00
NIROM 7.35 0.05 57.18 218.58 0 30.00
Team_Kanaloa -- -- -- -- -- --
owltonomous 75.62 79.87 0 0 0 0
Erau_Minion -- -- -- -- -- --
australis2 20.10 79.79 0 0 0 0
usyd_robotx 22.57 16.94 1753.87 0 0 0
auviitk -- -- -- -- -- --
uf 18.76 5.22 0 0 0 0
tang 86.84 17.81 0 0 0 0

Ranks Summary

Team Station keeping Wayfinding Perception Nav challenge Dock Scan and dock
mcgill_robotics 9 9 5 3 1 3
bumblebee 2 3 2 3 1 3
gt-marine-robotics 3 7 3 1 1 2
NIROM 1 1 4 2 1 1
Team_Kanaloa -- -- -- -- -- --
owltonomous 7 8 5 3 1 3
Erau_Minion -- -- -- -- -- --
australis2 5 6 5 3 1 3
usyd_robotx 6 4 1 3 1 3
auviitk -- -- -- -- -- --
uf 4 2 5 3 1 3
tang 8 5 5 3 1 3

Ranking

Team Total Ranking
NIROM 10 1
bumblebee 14 2
gt-marine-robotics 17 3
usyd_robotx 18 4
UF 18 5
australis2 23 6
tang 25 7
owltonomous 27 8
mcgill_robotics 30 9
auviitk -- -
Erau_Minion -- -
Team_Kanaloa -- -

Station keeping

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
mcgill_robotics 10.36 26.53 24.89 211.00 107.90 141.06
bumblebee 0.39 2.53 9.64 19.56 6.12 9.30
gt-marine-robotics 0.92 0.27 1.82 43.65 4.41 5.59
NIROM 0.22 0.42 2.31 24.49 5.56 11.09
Team_Kanaloa -- -- -- -- -- --
owltonomous 8.96 18.49 30.93 203.64 103.45 88.22
Erau_Minion -- -- -- -- -- --
australis2 0.14 0.49 7.96 72.46 13.27 26.26
usyd_robotx 0.94 1.05 3.50 76.59 17.30 36.05
auviitk -- -- -- -- -- --
uf 0.98 1.01 1.26 67.36 13.58 28.38
tang 10.23 26.40 24.59 211.05 107.91 140.85

Wayfinding

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
mcgill_robotics 41.00 75.44 150.27 46.50 82.65 83.35
bumblebee 0.10 21.71 31.54 0.12 15.84 17.19
gt-marine-robotics 40.81 75.25 150.20 46.30 82.86 83.34
NIROM 0 0.03 0.18 0.03 0.03 0.03
Team_Kanaloa -- -- -- -- -- --
owltonomous 41.00 75.45 150.27 46.50 82.62 83.36
Erau_Minion -- -- -- -- -- --
australis2 41.14 75.24 150.21 46.53 82.24 83.35
usyd_robotx 0.34 9.88 77.46 0.23 5.47 8.27
auviitk -- -- -- -- -- --
uf 0.40 0.54 28.62 0.50 0.76 0.52
tang 2.93 27.32 37.38 0.22 17.81 21.20

Perception

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
mcgill_robotics 0 0 0 0 0 0
bumblebee 10.18 12.56 33.19 12.22 16.76 23.87
gt-marine-robotics 0.81 6.59 37.21 0.80 8.77 35.18
NIROM 0.11 0.20 44.93 0.15 4.81 6.98
Team_Kanaloa -- -- -- -- -- --
owltonomous 0 0 0 0 0 0
Erau_Minion -- -- -- -- -- --
australis2 0 0 0 0 0 0
usyd_robotx 182.70 349.21 349.13 182.84 516.41 173.58
auviitk -- -- -- -- -- --
uf 0 0 0 0 0 0
tang 0 0 0 0 0 0

Navigation challenge

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
mcgill_robotics 0 0 0 0 0 0
bumblebee 0 0 0 0 0 0
gt-marine-robotics 0 402.6 0 169.11 0 0
NIROM 108.14 0 0 110.44 0 0
Team_Kanaloa -- -- -- -- -- --
owltonomous 0 0 0 0 0 0
Erau_Minion -- -- -- -- -- --
australis2 0 0 0 0 0 0
usyd_robotx 0 0 0 0 0 0
auviitk -- -- -- -- -- --
uf 0 0 0 0 0 0
tang 0 0 0 0 0 0

Dock

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
mcgill_robotics 0 0 0 0 0 0
bumblebee 0 0 0 0 0 0
gt-marine-robotics 10 10 10 10 10 10
NIROM 0 0 0 0 0 0
Team_Kanaloa -- -- -- -- -- --
owltonomous 0 0 0 0 0 0
Erau_Minion -- -- -- -- -- --
australis2 0 0 0 0 0 0
usyd_robotx 0 0 0 0 0 0
auviitk -- -- -- -- -- --
uf 0 0 0 0 0 0
tang 0 0 0 0 0 0

Scan and dock

Team Run 0 Run 1 Run 2 Run 3 Run 4 Run 5
mcgill_robotics 0 0 0 0 0 0
bumblebee 0 0 0 0 0 0
gt-marine-robotics 10 0 0 0 0 0
NIROM 10 10 0 10 0 0
Team_Kanaloa -- -- -- -- -- --
owltonomous 0 0 0 0 0 0
Erau_Minion -- -- -- -- -- --
australis2 0 0 0 0 0 0
usyd_robotx 0 0 0 0 0 0
auviitk -- -- -- -- -- --
uf 0 0 0 0 0 0
tang 0 0 0 0 0 0

Configuration Summary

We have been able to run the software for 1w of the 12 submitting teams for Phase 2. Below is a summary of the configurations (propulsion and sensors) being used by the teams.

Team Thrusters Cameras Lidars Lidar 1 Lidar 2 P3D
mcgill_robotics 2 3 1 16_beam NA Y
bumblebee 4 3 2 32_beam 32_beam Y
gt-marine-robotics 4 1 1 32_beam NA Y
NIROM 4 3 2 32_beam 16_beam Y
Team_Kanaloa 4 2 1 16_beam NA Y
owltonomous 2 2 1 32_beam NA N
Erau_Minion 2 3 2 32_beam 32_beam Y
australis2 3 2 1 32_beam NA Y
usyd_robotx 3 3 1 16_beam NA Y
auviitk 4 2 1 16_beam NA Y
uf 4 3 1 16_beam NA Y
tang 3 3 1 16_beam NA Y

Phase 1: Hello World

For Phase 1 of the VRX challenge teams were tasked with creating a video to demonstrate their ability to setup, provision and run the VRX virtual environment. This first step, "Hello World", showed that teams are well on their way to being able to submit full solutions for the next step, Phase 2: Dress Rehearsal.

Below are links to the 15 videos submitted by student and faculty teams from universities around the world.
We laughed, we cried, it was better than Cats.

Affiliation Team Video URL
Embry-Riddle Aeronautical University ERAU_Minion https://www.youtube.com/watch?v=e6H1vsERfyI
Flinders University australis2 https://vimeo.com/361766803
Indian Institute of Technology Kanpur auviitk https://drive.google.com/open?id=1zpgONDEywSaVN8yntFoMtZgUsg6AIejt
Georgia Institute of Technology gt-marine-robotics https://www.youtube.com/watch?v=mL9yQqX4VH0&feature=youtu.be
Korea Advanced Institute of Science and Technology (KAIST) NIROM https://www.youtube.com/watch?v=YqchycaU0aY
McGill University mcgill_robotics https://www.youtube.com/watch?v=jvBZCnQ0t9c
National University of Singapore bumblebee https://youtu.be/EGfL2FXSrlk
University of Sydney USYD_RobotX https://youtu.be/X4E_MOWlgK4
Florida Atlantic University team_owltonomous https://www.youtube.com/watch?v=WVvMmu9NS4A
National Taiwan University tang https://www.youtube.com/watch?v=Y45Wb5cffdM
University of Florida's Machine Intelligence Laboratory uf https://drive.google.com/file/d/1Ma8OEIQ1_oqBvMVylgv7mwJDRbWZ7k6e/view?usp=sharing
Osaka Prefecture University OUXT_Polaris https://youtu.be/SaizmxDic7E
Northwestern Polytechnical University Breaking Wave__NWPU https://drive.google.com/file/d/1kmRmKGPFeY1SejOj46DQwwFGLMErIqqG/view?usp=sharing
State Key Laboratory of Ocean Engineering, Shanghai Jiaotong University SKLOE https://www.youtube.com/watch?v=RTfHYwwzJeI
University of Hawaii at Manoa Kanaloa https://www.youtube.com/watch?v=aYpy9VSk5OI&t=17s
⚠️ **GitHub.com Fallback** ⚠️