Usability Report - mercal03/TINF21C_Team1_AAS-Server-Webclient GitHub Wiki
Changelog
| Version | Date | Author | Comment |
|---|---|---|---|
| 0.1 | 04.05.2023 | Martin Rittmann | Created, first draft, add introduction and summary |
| 1.0 | 10.05.2023 | Martin Rittmann | Add evaluation, results and conclusion |
Table of contents
Introduction
One of the main goals of our AAS-Webclient project is the improvement of the usability. The currently existing solutions are more technical and less user friendly. Therefore, this test report shall now summarize the usability approach and how it worked out by testing some of the main functionalities with a couple of test persons.
Executive Summary
The tests took place by the time the main functionalities as well as the graphical design were finalized. The test groups are made of people, who are familiar with digital twins and IT environments as well as external colleagues, who haven’t been in contact with this technology. The test itself takes about ten minutes per person and includes two parts. Firstly, the interviewed person can answer some questions about his/her experience in this area. In the second part there are three tasks that must be solved. The conducting person takes notes as well as the time the tested person takes and the thoughts that must be spoken out loud during the process of solving the task.
Methodology
In the following section there will be shown how the tests were set up and conducted.
Template (German)
Execution
Pre-Condition: The test person was not part of the development team and has no deep knowledge of the tested web-application. Step 1: The test person is asked the first questions and the conducting person notes all the answers. Step 2: The test person is shown the web-application and must solve the simple tasks without further explanation. During this process the test person shall tell all the thoughts coming up. Step 3: The conducting person notes the time, thoughts and possible problems that came up. Step 4: The test person is asked how difficult the tasks were on a scale from 1 to 5 and the conducting person notes all the answers including explanations. Step 5. The test person is asked if there is any more feedback regarding the design and usability.
Evaluation
Firstly, the test groups must be separated by experience to compare the length they needed to find the correct answers. Based on this we can evaluate if the user needs instructions or can intuitively solve the tasks even if there is no experience with the topic. Also, we will calculate the average time the participants needed to solve each task to see how fast and easy it is, to use the application. The thoughts and comments of the tested persons will be noted and used for future improvements of the application. Lastly, the personal assessment of the participants will be evaluated with the average of how usable they interpreted the website.
Results
Average time for each task
The participants are separated in three test groups. Group one has no experience with AAS-servers, digital twins or assets at all, test group has already heard of it, but not worked directly with it and test group three already worked with AAS before. The time is measured in seconds.
| Task | Test group 1 | Test group 2 | Test group 3 |
|---|---|---|---|
| 1 | 41 | 36 | 29 |
| 2 | 23 | 17 | 12 |
| 3 | 18 | 12 | 9 |
As expected, the less experienced group needed a bit more time to realize how the website works and where to add a new server. But after adding the server, the time for searching the item was similar in all test groups. The more experienced group was in parts a bit confused about the detailed information of the assets and why the manufacturing year is in the nameplate category. Some of them searched in the article information category. All in all, every person was able to solve the tasks fast after being set in front of the website without further explanation of the functionalities and what the goal of the website is.
Personal Assessment
In the following table there are the average assessments of the participants regarding the usability and difficulty of the tasks. Lower values represent high usability, intuitiveness as well as easiness of the task, higher values represent bad usability. The scale is from 1 to 5.
| Task | Test group 1 | Test group 2 | Test group 3 |
|---|---|---|---|
| 1 | 2,1 | 1,6 | 1,7 |
| 2 | 1,8 | 2,0 | 1,6 |
| 3 | 1,9 | 2,3 | 2,1 |
The personal assessment of the participants mirrors the results of the time evaluation before. The less experienced participants were able to solve the tasks very easy because of the intuitive design, which matches the classical construction of big websites in day-to-day usage for example the filter or search option on the left top side. The user is not confronted with all the information at once, which was the main goal of the project. Without seeing all the information at once, it is easy to find detailed information when specifically searching for it.
Comments
The comments of the participants reached from suggestions for improvements to satisfaction about the usability. Most suggestions were related to the informational categories in the asset detail view and some small bugfixes. The rework of the categories will be topic of future updates, but for now, this website is ready to launch for version 1.0 of project “AAS-Webclient”.
Conclusion
The tests revealed that the goal of this project is achieved as we created a webpage for digital twins, that can be used without further introduction or learning path. Even complete beginners can find digital twins and browse through the assets after adding a working AAS-server. It is therefore more usable and clearer than the solutions that existed before without sacrificing the amount of information, that can be found on the website.