5. Usability Testing - laurafo/Augmented-Flight-AVR GitHub Wiki

a. Preparation

For our usability test of the augmented flight application, we prepared an experiment guide and a subject code. Since the data and information we get from our subjects are treated anonymously, subjects can withdraw and delete their data via this code. The experiment guide contains demographic data to get a better understanding about our trial participants and what experience they have so far with augmented reality. It also contains several questions about our concept. Furthermore, we prepared two standardized questionnaires, the SUS (System Usability Scale) and the UEQ (User Experience Questionnaire).

b. Conduction

On January 3rd, 2022, we tested our application with 5 people. At the beginning, the demographic questions were answered and then each participant had five minutes to familiarize themselves with the AR application. During the five minutes, the participants were asked to think aloud so that we could take notes about it. Afterwards, several questions were answered about it. Lastly, after the two questionnaires were completed, the study was over.

c. Results

Experiment Guide

Five participants took part in our study, of which 40% were female and 60% male. The average age was 24.8 years and 80% have already tried an AR application, 20% use an AR app now and then.

The following questions, were all rated using a Likert scale of 1- 5, where 1 means "don't like it" and 5 means "really like it":

• How do you like the information that is shown to you? → Average score: 4.4
• How do you like the filter function? → Average score: 4.8
• If a pin interests you more, you can click on it to get more information. What do you think about this idea? → Average score: 5

Regarding the filter function, we asked another question: Would you rather filter by just one category or multiple categories at once and why? → 40% would like to be able to apply only one filter, 60% prefer more.

Another question we asked was what other information participants would like to see:

• Pictures/videos of the individual pins
• Ratings
• Booking the attractions and hotels directly through the application

The final question was "Now that you're a little more familiar with the concept of the app, we'd be interested to hear what you think of the idea and if you'd like to use it on a flight." All participants would like to use our application on a flight. The possibility to fold the applications and just enjoy the view was very well received. Two of the participants look at the cities and countries they fly over before each flight and would love to be able to use this concept in real life. In addition, everyone thinks the concept is very nicely laid out and doesn't interfere with the view out of the airplane window, as the pins are discreet, making it a perfect combination of view and information.

System Usability Scale (SUS)

The overall usability was defined by this standardized Questionnaire. The results can be transferred into a score that corresponds with a certain level of usability. Our application has achieved an SUS Score of 92, which means that the usability of Augmented Flight can be defined as "excellent".

User Experience Questionnaire (UEQ)

The UEQ shows the pragmatic and hedonic quality, as well as the overall user experience. Again, our application is described as "excellent".