Project – Pianololens

Team Members

Julia Malsam, Alina Megos, Josef Huber

Idea

We want to create an augmented reality HoloLens 2 application, to learn how to play the piano.

Motivation

Learning how to play the piano is a tough undertaking. Many people start euphorically learning it but quit after the first couple of days. One of our team members speaks out of experience. Especially learning the piano on your own without any teacher takes a lot of effort and willpower. We want to make learning the Piano easier and see a great opportunity in using augmented reality and HoloLens 2 for it.

Proposed Outcome

The proposed outcome should be a proof of concept for a HoloLens 2 application which should help you to learn and play the piano. The application should contain at least one piano melody which can be learned and played through with augmented reality guidance.

Requirements for the project

General Requirements:

1. Usage of a real Piano or Keyboard to play on
2. Sitting position on a chair in a room with good lightning
3. Marker on the Piano for the right positioning of the AR Hints
4. Usage of HoloLens 2 for the application
5. Quiet environment to concentrate

Team Requirements

1. Team Member who knows how to play and learn the Piano – Julia, Alina
2. Team Member who can create 3D Models – Julia
3. Team Member who knows how to program – Alina, Joe
4. Team Member who is familiar with Unity – Joe
5. Team Member who is not afraid to try new things and always keeps learning – Julia, Alina, Joe

Software Requirements

• Unity 3D
• Microsoft Mixed Reality Toolkit
• Git + GitHub
• Visual Studio / IntelliJ Student
• 3D Software – Blender, Cinema 4D, Rhino
• Illustrator/Figma – Conception

Hardware Requirements

• Piano/Keyboard
• HoloLens 2
• Hands
• Chair/Table/Room/Environment

Interaction

The interaction should be done simply by playing the real piano. The user plays the piano with his real hands. Over the piano will be augmented reality hints displayed as bars with different length and color, that will indicate which note should be played next and for how long. Visually it should look like the hints will overlay the piano keys and “fly in” from the front of the keys. As soon as the bar reaches the key, the key gets highlighted - The key should then be played by the trainee as long as the bar flows into it. During this event the bar shrinks. If the Trainee misses the note, the HoloLens will visually show that the note is missed. (Additionally, it could play an error audio sound.)

Educational wise, the trainee should have the following possibilities:

• to slow down the melody and train it first in slower pace
• Pause the melody
• Restart the melody
• Skip back and forth in the timeline in chapters to train certain passages again
• Have a menu to select different melodies in different difficulties
• Learn hand by hand – first left hand, then right hand, then both
• Pause the melody if the hands are taken from the keys

This control options should be done by:

• preferably voice command – simply saying:” Retry this chapter” or “Slow Down”
• or via an augmented menu which can be brought up by looking at your wrist for a certain time and then selecting the option via hand tracking.

Reference

Moving pictures speak more than 1000 words. Here is a good reference video on how this learning technique currently looks like in 2D Video. https://www.youtube.com/watch?v=kLeRxT9RcII

2D Video Problems

The problem here is, that the trainee constantly has to switch between viewing at the screen and the piano keys. Additionally, the trainee constantly has to switch his hands between piano keys and the computer mouse, to stop the video, skip some seconds back or change the speed. We want to eliminate all of that through our augmented reality piano learning application – Let’s go!

Outline of project

1. Analyze Piano Applications
2. Expert/User Interviews and Observation
3. Personas / User Journey / Storyboards
4. Requirements Definition
5. Wireframes in Figma
6. First MVP Concept in Unity
7. Usability Testing
8. Refinement Concept Unity – Code Freeze
9. Documentation
10. Presentation – Demo or Image Video/Wizard of Oz?

Outline of implementation

1. Getting familiar with unity and setting up a first project.
2. Getting familiar with Microsoft Mixed Reality Toolkit and incusing it in the project.
3. Display a button in a HoloLens 2 Application an make it touchable – including touch events.
4. Let different buttons spawn and fly towards the user – user needs to be able touch those easily.
5. Organize buttons in a keyboard layout and place them in relation to a marker placed on a piano.
6. Let buttons fly in synchronized to a piano melody.
7. Adjust length of the buttons according to how long the note should be played.
8. Make a disappear animation of the button when key is played.
9. Include error visuals.
10. Include options to slow down and restart.
11. Include skip forward backward option.
12. Include Menu via Menu via speech.

Open Questions

1. Should it be in the end a fully functionable prototype
2. Or should it be an MVP Prototype, with focus on a completely thought trough concept with maybe a Wizard of Oz Image video at the end?