Home - ImmersiveStorytelling/DocumentationMasterclass GitHub Wiki

In this project we try to get more understanding about Virtual Reality and how to create stories as immersive as possible. As students from AP Hogeschool Electronics-ICT, we joined a masterclass “Show and Tell” where master students from Royal Academy of Fine Arts Antwerp and Sint-Lucas School of Arts were given the assignment to create a creative immersive story in the virtual reality.

Being part of a research cooperation “Immersive Storytelling” from our schools, our job was to research how to work with 360° videos in Unity and gather information both technically and psychology to create a deep immersion in a virtual reality story.

Four groups were made among the master students who were responsible for creating a story which included constructing decors, acting and filming. Our job, as the technical team, was to experiment(Experiments) to see if certain ideas were possible, aiding with the recording and assemble all the shots in that way the story could be told the way it was immersive.

From the four groups where thus four stories created, from which we will choose one to analyze for further research. A quick list of the projects and ideas:

  • Room Cleanup: in this project the user looks around in a old dirty abandoned room. The more the user looks around, the cleaner the room will get until it’s completely cleaned (without the user seeing the changes happening) as if someone cleaned and inhabited the room.
  • Coffee Ritual: in this project the students played with the same room in different scenarios where the user sits in the middle of the room and coffee is getting prepared in a very absurd way for the user to drink. For each action to make coffee is done in a different scene and room, but the user won’t see the change of rooms. After the last scene is played, the scenario will be played infinite further, where the ‘first’ scene follows precisely on the last one.
  • Doctor’s Office: here you are subject to certain vague experiments in the virtual reality world, with impulses given from the real world outside. You wake up in a strange room covered with plastic and a strange doctor and his assistant. Then the doctor will take certain actions to the subject, which includes impulses from the real world. Depending on how you react to certain things, the story will adapt.
  • Black Room: in this room everything is black where strange and vague things happen. The room fills with smoke, lights up, goes completely dark, people start walking on the ceiling, puppets are everywhere, strange things happen wherever you look so that you get really disorientated and confused.

The project we chose is Black Room, since it was the most variant and difficult project to put together. Further research of this on the Black Room Wiki page.

Virtual Reality with 360° videos (philosophy & challenges)

As people always have been obsessive with telling stories, the world has known countless ways of telling stories. This goes from mouth to mouth stories, to stories in books, pictures, movies, performances, dances, art works, poetics and much more. Many of them have been there as long as humans themselves, other ways are still very young. We always keep searching for different ways of telling stories, because this enables us to tell a story in a way it could not have been told before.

In this age where technology became so important and widespread, it also made it possible for new ways of creating stories, where one of the newest ones certainly is Virtual Reality. It is still a technology not widely explored and still has a long way to go, but despite this it is really fast growing and is much used today and will be more in the future once it becomes more accessible for consumers.

Virtual Reality is a new and very separate way of telling stories from others, which brings along a few challenges as for making it possible and also how to tell a story that is immersive. Just creating a story won’t work for VR since the user has a certain freedom in the VR by being able to look wherever he/she wants to. The question that first came up, was then how could we get the user’s attention to tell a story which can’t be forced (unlike a book or movie). Many other challenges soon emerged:

  • Telling a story that happens all around and not just in front of the user
  • Distracting the user to look away from something, of towards something
  • What to do when the user doesn’t look to the proper direction
  • How and when to apply changes in the videos without the user seeing them
  • Apply proper sounds to the story which improves the immersion of the story
  • …

Technical challenges

Trying to tell a story in VR as immersive as possible, we had to adapt ourselves to a way of working where the story could be told that way it fits the user. Since we cannot force the user to look at something specific, we had to make it possible that the story itself knew how to react to the user's behavior. So the first challenge was how to get to know the user reactions, by for example knowing the position or direction of sight of the user, and next how to use this data for the story to react to, for example change to next shot after something is seen.

Another big challenge was how to switch between two shots without the user noticing the video changed shots. This brought many difficulties. First of all, we had to make sure that when the one shot changed to the next, the user didn´t get to see this wherever he/she was looking at. This required so that when the shot had to change, that at that moment the shot (where the user is looking at) was exactly the same as the beginning of the next shot. To accomplish this there may be absolutely no movement, appearing/disappearing objects/persons, change of lightning, different settings of the camera and certainly no displacement of the camera itself that records the different shots (unless otherwise intended).

Once we had the recordings, we had to make it possible for the video to be played inside the virtual reality. For converting the recordings to a proper video format we could use in Unity, we used ffmpeg to divide the video in frames, so then we could use these frames with HPV-Creator to create a video format which could be played in Unity.

Next challenges then were:

Technology used

To get this project to a good ending, we had to use specific technology that meets our requirements.

  • Virtual Reality set: HTC Vive
  • Virtual Reality development software: Unity
    • Since the stories can’t react the same on different users, we needed to make the program possible to react to user input by programming certain actions. Therefore we used C# in Unity, which makes it possible to attach Scripts to game objects that handles the user input and adapts to it.
  • Camera: Garmin 360°
  • Converting programs:
  • Starting code:
    • Spheres: given to us by Tom Peeters