Methodology

Overview

The latest version of the game will be shown to multiple users to gather feedback on the prologue, epilogue, music, and general feel of the sounds/visuals. The user will be briefed about the current stage of development, however they will not be told about the genre or story of the game. The user will have a list of tasks to complete where they will describe their thoughts out loud while completing them. After a task is complete, an open ended question may be asked such as "How does the current screen make you feel?" if the question was not answered by the user thinking out loud. After the user has completed all the tasks they will be given a short questionnaire to fill out for primarily quantitative data.

Justification

The story and music are integral to creating the world and the atmosphere.

Conveying a story within a few screens can be tricky, and as the game designers know the story in detail, it is possible that they are unaware of missing or confusing story elements. Through user testing we can ask the user to describe the story (without any prompts) to determine whether the story that we want to tell is clear.

When creating atmosphere for the game, the music and visuals can be subjective, therefore it is crucial that feedback is gathered from the potential target audience to check that the correct atmosphere/mood has been set.

Testing Process

User Brief

Before the user testing commences the user will be given an explanation about how "thinking aloud" testing will work and the current stage of game development. The games genre and target audience will not be revealed to the user, however for ethical reasons the users will be notified that the game is for mature adults and may contain disturbing scenes. Due to the current state of the game, the user testing will be run as more of a stop-and-start approach, where the user will complete one task and then be asked to talk through their thoughts. The user will be notified that the observer will remain silent while the task is being performed but if they have any question then they can ask the observer.

User Tasks

Title screen

The game will be loaded on the title screen where the user will be asked to describe their current thoughts on the game. If the user is unsure of what to talk about then the following questions will be prompted:

• From your first impressions of the game, what genre do you feel this game is?
• From your first impressions of the game, who do you feel its target audience is?
• Describe what you would expect to happen when you click each menu button.
• Describe any feelings you feel from looking the title screen.
• Describe any feelings you feel from the background music.
• Describe what you think the images in the background might be.
• Are there any additional buttons you would want on the menu screen?

Prologue and pause screen

The user will be given the task to start a new game and finish the intro scene. Once the prologue has been completed the user will be given the task to pause the game; if the user has trouble pausing the game then this will be noted. If the user did not answer any of the following questions then they will be prompted once the game is paused:

• Describe what you feel is the story of the game.
• Describe the objective of the game.
• (if required) Another part of the story was XXXX, how clear do you feel this was from the prologue?
• Describe what feelings you felt from the prologue
• How intuitive/smooth did you find progressing through the screens in the dialogue? Do you have any suggestions on how to make it more intuitive/smooth?
• How would you describe the pacing of the prologue?
• Describe what you would expect to happen when you click each menu button.
• Are there any additional buttons you would want on the pause screen?

The Game

As the purpose the this user testing is not to test game mechanics, the user will be given brief instructions on how to play the game. The user will be then given the task to reach an ending in the game (either game-over or epilogue). If the user reaches the game-over ending then they will continue with the tasks in the "Game Over" section. If the user reaches the epilogue then they will not be interrupted until the epilogue has been concluded.

Game over

When they reach the game over screen they will be prompted with the following questions:

• Describe the feelings you have on the game over screen.
• Describe what feelings you have from the music in the background.
• Describe what you would expect to happen when you click each menu button.
• Are there any additional buttons or features you would want on the game over screen?

Epilogue

After the epilogue concludes the user will be prompted with the following questions (if they have not already answered during thinking out loud):

• Describe what you feel happened during the epilogue.
• (if required) Another part of the story was XXXX, how clear do you feel this was from the epilogue?
• Describe what feelings you felt from the epilogue
• How would you describe the pacing of the epilogue?

Other Music

If the user has not talked about the following questions then they will be asked once both endings have been reached. If the user does not recall the music for that section then they will be brought to the location where the music plays.

• How did the music within the safehouse make you feel?
• How did the music for the final level make you feel?

Small Questionnaire

A small questionnaire will be given to the user to complete after all tasks are complete. These questions will contain a Likert scale.

• After the prologue, I was intrigued in the story line
• The story was clear and understandable
• The game objective was clear from the prologue
• The game was scary/horrifying at times
• The game has a clear and consistent art style (main title, prologue, and epilogue only)
• The games conclusion was satisfying

Testing

Participant #1: Significant observations and feedback

Title screen

• Had issues reading the title text with the different height letter positioning, also didn't really understand why it was like that.
• Thought the game was a retro (due to the pixel art) and dark/creepy (due to the music).
• Thought the game was for 16-40 year olds but didn't look particular scary.
• Felt calm from the title screen, not that excited to play it.
• Felt calm from the music, didn't find the music annoying at all. Liked the looping.
• Correctly identified clouds in the background, thought the light could be a fire.
• The description of their expectations for what each button would do lined up with the current functionality
• Would like to see a load button as they get frustrated with games they can't save/load.

Prologue

• Tried to progress the dialogue by clicking the mouse before realising it is the spacebar.
• Some of the wording felt clunky and didn't flow well (they listed dialogue they felt was clunky)
• Some of the wording was unclear (they listed dialogue they felt was unclear)
• Felt that the art style between the title screen and prologue was very similar, which made it feel inconsistent when the screens weren't animated like the title screen. Overall felt that each screen needed to be more animated (they listed screens in particular that they wanted more animated).
• Wanted the larger dialogue boxes to be split into two to create more suspense.
• Was confused to why the enemy was referred to as a shadow crawler but then the next screen called them demon zombies. Felt that demon zombies lost a bit of the scary/unknown feeling that the shadow crawler was giving them.
• BUG: game crashed when not progressing a dialogue box for 10 minutes, and then trying to progress it.
• Found it very jarring going from the dark prologue to the bright green game environment.
• Text was on the very far edge of the dialogue box making it uncomfortable to read.
• They didn't understand what a safehouse was as it was not mentioned in the prologue.
• Description of what they thought the game objective was: "I need to get to the big ball of light that was in the title screen. I am a firefly. There will probably be monsters chasing me. I understand what the fireflies are about, the darkness, and that I'm trying to find safety."
• Feelings from the prologue: "Liked it, sounded like a cool premise. Fear of the dark. The unknown. Didn't feel much like a horror but rather than a escape the monster game (but could end up being a horror). Feels like it is going to be a primarily adventure like game with light and dark being a mechanic."
• Suggestions to make it smoother: "Adding multiple keys and mouse to progress would help. Going back could be nice as well by pressing the left key. If I have to keep playing the game over than don't want to have to progress through the prologue each time, would like a skip button."

Pause Menu

• BUG: game crashed when not clicking a button for 10 minutes, and then trying to click one.
• The description of their expectations for what each button would do lined up with the current functionality
• Would like to control it with the keyboard and spacebar to select a button. Found it strange that the prologue you could only use the keyboard and then the pause menu could only use the mouse.
• Would like to see a save and a load button as they get frustrated with games they can't save/load.
• Liked that it was clear where each exit button would take them.

Game Over Screen

• Loved the music. Felt the music was mocking you for dying (going from silence to funeral march like music). Sounds like pipe orgrans, it felt spooky.
• Didn't think the game over text matched the rest of the game (felt that both the colour and the font was not used before this screen).
• Liked the firefly lights in the background, however was confused why they are actual fireflies now instead of fireflies being the people who run to the safe haven (note this feature was removed after user testing)
• Felt that it was strange to swap from keyboard back to mouse again for this screen.
• Wanted their to be a fade to black after the player died to symbolise the darkness getting them.
• Thought that the restart button will go back to after the prologue. This is different to the current functionality where it takes you back to before the prologue.
• Wasn't sure what the "Exit" button would do. If it takes you back to the desktop then feels that it should be "Exit to desktop".
• Didn't feel the gameover screen needed a "Exit" button, just a return to menu button.

Epilogue

• Was very confused about the dialogue talking about telling the others and heading home, but then the person gets into the boat and sails away instead.
• Loved the light over the water image.
• Felt that the dialogue boxes needed to be broken into multiple smaller boxes. This was to let the player feel more of a victory before the plot twist.
• Had issue with some of the logic of the dialogue (listed which dialogue they had issues with).
• Some of the wording felt clunky and didn't flow well (they listed dialogue they felt was clunky).
• Wanted to see more animations.
• Wondered why the shadow crawler was now called a shadow instead.
• Prologue ends abruptly and returns to the main menu. Suggested that it should fade to black at the end and have a sound of a boat crunching as the shadow crawler destroys the boat.
• Liked that it went back to the menu screen. They didn't feel that it needed a "The End" screen or anything like that.
• Describing what they thought happened: "I made it to the ocean, for some reason I felt that safety was beyond the ocean. I jump on a boat and sail towards the light. The shadows come at me and then I'm gone (consumed)."
• Feelings felt in the prologue: "depressing, yay I won, succeeded, got there, then I died, false hope."
• Preferred the pacing of the prologue, as the epilogue seemed to progress too quickly. Would of liked the going into the glow shots to be longer to "Draw out the success before crushing the persons hope".

Other Music

• Feelings about the safehouse music: "Cheerful, very dark. Cheerful take on the main theme."
• Feelings on the final level music: "Felt more like a boss fight. Sounds more like a video game. Not as dark as the other music." (note that the interviewer forgot to mention that this music was meant for a boss battle).

Participant #2: Significant observations and feedback

Interviewer: Renée

Tested commit: e12f659

Title screen

• The second letter "I" is a bit hard to see
• They would like to have the option to use arrow keys to navigate the menu

Prologue

• Having to switch to the spacebar [from the mouse in the title screen] to advance the diaogue is inconsistent.
• They were not prepared for the beginning of the first level and was attacked the moment the game loaded. They suggested that showing a card that says "Level 1" would help.
• In their second playthough, they wanted an option to skip the prologue.

Pause Menu

• They liked how the music continued through the pause screen.

Epilogue

• The test subject did not realise that this was an epilogue until they were told.
• "That's an abrupt spot to finish it, because it sets up a plot thread that does not get resolved by either either a cutscene or more gameplay. It shouldn't be used to introduce new things."

Music

• Was surprised that the music stopped at the settings screen and didn't play through.
• Completed our soundtrack.

Other comments

• "A lot of the transitions in the game need to be smoothed out."
• The game crashed many times during testing around the safehouse. I asked my tester to run directly to the safehouse without interacting with any enemies to reduce the likelihood of this happening.

Participant #3: Significant Observations & Feedback

Title Screen

• From first impressions were it being an adventure/horror genre.
• was curious and intrigued by the art style and the music

Prologue

• very inciteful
• a deeper understanding of the scenario/situation
• the art style matched the title screen (the pixels and colours)
• was confused on how to progress dialogue - would like an indicator to press space to continue

Pause Menu

• "Does the job"

Epilogue

• felt a bit rushed - but it was because there isn't the main game story from the gameplay
• however, it made you feel safe, then obviously the dialogue and then the shadow caught you off guard

Game Over

• the music sounds "very dead-esque"
• The lighting effect looks like fireflies leaving
• the font gives off grim reaper vibes

Music

• The game over music was very fitting
• Safe house music made you feel safe
• Final Boss Battle: very fast paced. knew it was boss music without being told

Other Comments

• was confused on how to play the game when they first finished the prologue - mentioned adding a "how to play"/tutorial on the title screen

Participant #4: Significant Observations & Feedback

Title Screen

• Lamp's fire doesn't look like it's emitting light.
• The shape of the cloud is abnormal which is difficult to recognise especially as a pixel art.
• Too many components are moving in the title.
• Overall background is confusing especially in relation to the title 'Fireflies'

Prologue

• No indicator to progress dialogue
• Dialogue felt a bit long to get through, too many unnecessary details.
• Doesn't really explain the objective of the game.
• Fits the theme of the game.

Pause Menu

• Nothing special

Epilogue

• Similar to prologue, no indicator to progress dialogue.
• Confusing.

Game Over

• Wasn't expecting the prologue to be repeated on restarting the game.

Music

• Retro effect is nice, but melody sounds a bit too cute for a spooky game.
• Safehouse music makes you feel safe and is a nice variation of the title screen music.
• Final boss music was odd that it didn't have retro sound similar to title screen and safe house music. "A lot of it is in a high register and doesn’t have any bass sound so it doesn’t have much impact or dramatic flair. "

Participant #5: Significant Observations & Feedback

Title Screen

• Shape of the cloud looks like smoke
• From the title screen alone the genre looks like it's horror, RPG or fantasy
• Target audience was thought to be around 8 years.
• The glowing thing looks like lots and lots of fireflies
• Sounds and mute button could be useful addition

Prologue

• Bit long, would liked skip button on repeat playthroughs
• Sense of urgency and fear from the character
• Would liked auto playing dialogue
• Liked art style

Pause Menu

• Would liked control mapping

Epilogue

• Very confused about ending, was going to a safe place but now on a boat?
• Not a very satisfying ending
• A lot more rushed (more story cramming) than the prologue

Game Over

• Very "You Lost! :-(" kind of feeling

Music

• Loved the intro music
• Safehouse music felt happy and safe
• Game over music feels like inside a cathedral with vampires

Survey results

The average survey results for the testers (rounded to the nearest option):

After the prologue, I was intrigued in the story line: Strongly Agree

The story was clear and understandable: Strongly Agree

The game objective was clear from the prologue: Agree

The game was scary/horrifying at times: Disagree

The game has a clear and consistent art style (main title, prologue, and epilogue only): Agree

The games conclusion was satisfying: Neither Agree or Disagree

Evaluation of Results

Common Issues & General Options for improvement:

• Dialogue confusing or didn't flow in some parts of the story scenes. Some dialogue boxes could be split up into two dialogue to either create suspense (prologue) or sense of fulfilment before the plot twist (epilogue).
• More animations and transitions in the story scenes.
• Fade transition into game over screen. (Let the player realise they are dead too.)
• Save/Load game menu items.
• An option to skip the prologue scene.
• Scenes end suddenly and therefore feel jarring.
• Game doesn't seem scary.
• More consistency with mouse or keyboard interactions for the story scenes and menus. Preferable enable both.
• Dialogue text is too far to the left of the dialogue box.
• Possible bug related to time, where after a certain amount of time elapses with no input crashes the game.
• Inconsistency with the game over text.
• Didn't feel much like an epilogue, needs more conclusion. Possibly a fade to black and then a boat crunch noise as the shadow crawler hits the boat.

Currently Actionable Improvements:

• Add a listener for the escape key that lets the user skip the prologue scene.
• Modify some of the dialogue and scene transitions to make it feel smoother.
• Add some new dialogue to explain unclear story features.
• Remove some dialogue that was confusing the user.
• Reduce the width of the dialogue boxes so that the text doesn't feel so far to the left side.
• Change the game over font to be similar to the title screen and change the colour from red to orange.