Methodology

Overview

The latest version of the game will be shown to multiple users to gather feedback on the mid-level cutscenes and NPCS. The user will be briefed about the current stage of development, however they will not be told about the genre or story of the game. The user will have a list of tasks to complete where they will describe their thoughts out loud while completing them. After a task is complete, an open ended question may be asked such as "How does the current screen make you feel?" if the question was not answered by the user thinking out loud. After the user has completed all the tasks they will be given a short questionnaire to fill out for primarily quantitative data.

Justification

Mid-level cutscenes

The story is integral to creating the world and the atmosphere.

Conveying a story within a few screens can be tricky, and as the game designers know the story in detail, it is possible that they are unaware of missing or confusing story elements. Through user testing we can ask the user to describe the story (without any prompts) to determine whether the story that we want to tell is clear.

Friendly NPCs

The point of friendly NPCs is to create a more immersive world through story telling and interacting with the environment. As with the mid-level cutscenes, conveying a story within a few dialogue boxes can be tricky, and this is where user testing comes in. From the user testing we seek to:

• Determine whether the story points made by each friendly NPCs are clear
• How an NPC adds or subtracts from the world immersion or the mood we are aiming for.

As friendly NPCs are a new interactive component to the game we also need to make sure the user does not get frustrated with them. We will be checking:

• Is it obvious that the user can interact with the friendly NPCs
• Is it obvious how the user can interact with the friendly NPCs
• Are there any friendly NPCs that have too much or too little dialogue
• Are there too little or too many friendly NPCs

Testing Process

User Brief

Before the user testing commences the user will be given an explanation about how "thinking aloud" testing will work and the current stage of game development. The games genre and target audience will be explained to the user, and the users will be notified that the game may contain disturbing scenes. The user will be notified that the observer will remain silent while the task is being performed but if they have any question then they can ask the observer.

User Tasks (Qualitative testing)

Task 1 - Talk to some of the people around town

Questions asked at the end of the task if they are not already answered during observations:

• (If they missed an NPC) Did you notice any other NPCs that you did not talk to? (If yes) why did you choose not to talk to them?
• Explain some feeling you had when interacting with particular NPCs.
• State what you think each NPCs role and personality might be. Note these aspects are not explained in the game so it will be pure speculation from the tester.
• Suggest one thing for each NPCs dialogue that might make it easier to understand.
• Suggest one thing for each NPCs dialogue that might add to the world immersion or horror theme.
• Suggest one change to how the tutorial on the game's controls is currently explained
• How did you find the controls used to interact with the NPCs?

Task 2 - Progress to the second level of the game

Questions asked at the end of the task if they are not already answered during observations:

• All the questions from task one regarding the Hurt firefly NPC
• Explain how the player has progressed through the game so far (for example "the character walked through a desert and found an oasis")
• What do you feel is the objective of the next level (for example "find the treasure")?
• Suggest one thing for the in-between level cutscene that might make it easier to understand.
• Suggest one thing for the in-between level cutscene that might add to the world immersion or horror theme.

Task 3 - Progress to the third level of the game

Questions asked at the end of the task if they are not already answered during observations:

• All the questions from task one regarding the dead body NPC
• Explain how the player has progressed through the game so far (for example "the character walked through a desert and found an oasis")
• What do you feel is the objective of the next level (for example "find the treasure")?
• Suggest one thing for the in-between level cutscene that might make it easier to understand.
• Suggest one thing for the in-between level cutscene that might add to the world immersion or horror theme.

Task 4 - Finish the game and return back to the main menu

Questions asked at the end of the task if they are not already answered during observations:

• Explain the overall story and how the player progressed throughout the game.
• Explain the overall feelings that you felt through the story scenes
• Would you like to see more NPCs in the game? (If yes) Any particular character personalities or stories you would like added?

Small Questionnaire (Quantitative testing)

A small questionnaire will be given to the user to complete after all tasks are complete. These questions will contain a Likert scale.

• I understood what the games controls were
• It was clear who were the friendly NPCs that I could talk to
• Interacting with friendly NPCs was intuitive and smooth
• The dialogue for the friendly NPCs was clunky and hard to understand
• I liked the variety of different friendly NPCs
• There were not enough friendly NPCs in the game
• I found the friendly NPCs interesting
• I feel that the friendly NPCs added to the immersion of the world
• I learnt more about the story through interacting with the friendly NPCs
• The story cutscenes in-between levels helped me understand the games objective and progression
• The dialogue for the cutscenes was clunky and hard to understand
• The story elements (NPCs and cutscenes) were consistent in their world building, i.e. didn't contradict themself within the world the game has created.
• The story elements (NPCs and cutscenes) consistently added to the horror theme of the game

Testing

• We tested commit 5bade138f3cff3df3d9b32c397cbbe93aef8f5ef

Participant #1: Significant observations and feedback

Pilot NPC

• He looks suspicious, I feel that I shouldn't trust him.
• Didn't feel that he added anything to the story? Was he meant to help at all, or should I have gotten somthing from him? Why is the pilot there?
• Suggestion: He could possible talk about health/healing and give a bandage or something and say how to use bandages, as I had no idea how to use bandages. Something like "Hey here is my medkit, press 1 to use it".
• Suggestion: Maybe he could be a firefly that is also making the journey and then we find his dead body later in the game.
• Didn't feel like his dialogue or looks added much to the horror themes

Tutorial NPC

• He seems to be a helpful old man
• Didn't mind him telling me about how to play the game, as it had a retro game feel to it
• Didn't feel like his dialogue or looks added much to the horror themes
• He starts off by calling me "young man", why do I have to be a man?
• Sometimes read the keys as words rather than keyboard keys. Could you have the image of the keys (like WASD and Enter).
• Suggestion: Change the dialogue to be a little more in the horror theme. Maybe something like "You're by yourself, that's good... that way you won't lose the people you love" or "A group of us tried once... but by the time it was just me... well I... he looks away... anyway goodluck out there."
• Suggestion: Make the looks a bit more rough/raggy/haunted, they (refering to the pilot as well) don't look scared. He looks very clean and wealthy which seems jarring with the apocolyptic themes. At the moment it feels more like a mystery game rather than a horror.

Level 1 to level 2 cutscene:

• Love the images of the forest with the glow in the background
• Not sure why it said "keep following the fireflies", should it be the path of the fireflies? As I don't remember following anyone in particular.
• Thought that the safehouse in the cutscene referred to the one they are currently in, but understood that the goal of level 2 was to get to the next safehouse.
• Suggestion: Having longer shadows or darker shadows on the trees might make it more spookier.

Injured firefly NPC

• Speech bubble helped me know that I could talk to him
• Got to sit there and watch someones last moment as they bleed to death. Felt disturbed, probably one of the darkest things so far.
• Didn't like the "greedy bugger" at the end of the speech. Sounded like a joke, but seemed to take away from the horror (if that was what was wanted).

Dead NPC

• Thought the person was a monster, I didn't think it was a person or something I should get close to. I'm trying to steer clear of everything, so thought I would avoid what I thought was an antlion monster. Only reason I talked to it was the speech bubble.
• No picture with the person. If you want more horror then having a face of a dead person there would be really disturbing.
• Didn't mind the aside dialogue (i.e. the asterisks like old school games)
• Suggestion: Why do I care about this body? I feel I would care more if it was related to the story in some way, maybe someone I met earlier in the game? Maybe a friend/relative of mine? Maybe I could recognise something about their clothing and say how that was my best friend Johnny's Tshirt.

Level 2 to level 3 cutscene:

• Love the blue ocean and that the glow has gotten bigger
• Felt like the next level was going to the beach but was confused that the level wasn't a beach.
• Felt the cutscene was hopeful

Other

• Was confused why there is so much space in the tutorial area if there is no where to go or talk to. The tester walked up towards the city buildings looking for more NPCs or interactive elements.
• Didn't think that the game needed more NPCs as the game is very difficult so I'm more focused on running past monsters. Also didn't expect to find NPCs in the levels due to the same reasons.

Participant #2: Significant observations and feedback

• Tested by: Renée on Mon 04/10/21

Tutorial NPC

• Instructions on how to throw and reload rocks was not mentioned.
• Didn't have any feelings towards this NPC as they didn't have much of a personality.

Pilot NPC

• They felt inclined not to like the pilot because of how he boasts of his achievements but at the same time felt that his advice on staying safe was sound.
• His personality therefore seems inconsistent.

Injured firefly NPC

• They found the 'selfishness' comment funny, but maybe too light-hearted for a person who is dying.

Dead NPC

• Didn't have any feelings towards this NPC as they didn't have much of a personality.
• The visual design of the NPC reminded the tester of corpses from Among Us, not necessarily in a good or bad way.

Inter-level cutscenes:

• Cutscenes used to good effect in telling the story in stages. Please fix punctuation errors.

Other

• A lot of the NPCs have too much personality for this to be a horror game.
• They're too amusing, especially for such a small game to only have non-serious characters.
• The game needs more NPCs, especially in the second level and safehouse.
• They liked how NPCs were denoted with speech bubbles to encourage players to approach them, but it was less clear how an enemy was marked.
• There was a sustained feeling of uncertainty throughout the game that successfully conveyed the horror genre of the game. Despite the simplicity of the game, it felt "threatening".

Participant #3: Significant observations and feedback

Pilot NPC

• Added hints for what's to come
• Liked the pilot overall as it adds mystery to that particular character

Tutorial NPC

• Tutorial dump, lots of keys to remember
• Perhaps having different people for different controls
• Controls to trigger dialogue are good, except when switching from e to spacebar key

Level 1 to level 2 cutscene:

• Add more emotional connection to the cutscene instead of only, "I must go to this place etc."
• Loading indictor for changing between scenes.
• Elaborate on the previous events when transversing the city. A bit more on what's going on inside the character's head. e.g. guilt of leaving he person behind.

Injured firefly NPC

• Dialogue was funny, comic relief
• Suggested dialogue "Leave me here, save yourself"

Dead NPC

• Completely missed dead npc
• Due to the amount of enemies on the field.

Participant #4: Significant observations and feedback

Pilot NPC

• Looks confident and mysterious
• Doesn't add much to the game in the starting area. Would have liked to see him again further in the game.

Tutorial NPC

• Portrait looks too clean and happy for the game.
• Dialogue feels off and too gamey.

Level 1 to level 2 cutscene:

• Nice illustration.
• The house appearing out of nowhere when there were no changes in the tree was sudden and weird.

Injured firefly NPC

• Horrifying dialogue portrait. Could use some polishing
• Thematic NPC for the game.

Dead NPC

• Didn't know it was something they could talk to, until it was pointed out.
• Similar to injured npc, but wasn't sure what the sprite was meant to resemble.

Survey results

The average survey results for the testers (rounded to the nearest option):

I understood what the games controls were: Agree

It was clear who were the friendly NPCs that I could talk to: Agree

Interacting with friendly NPCs was intuitive and smooth: Strongly Agree

The dialogue for the friendly NPCs was clunky and hard to understand: Disagree

I liked the variety of different friendly NPCs: Neither Agree or Disagree

There were not enough friendly NPCs in the game: Agree

I found the friendly NPCs interesting: Neither Agree or Disagree

I feel that the friendly NPCs added to the immersion of the world: Agree

I learnt more about the story through interacting with the friendly NPCs: Agree

The story cutscenes in-between levels helped me understand the games objective and progression: Strongly Agree

The dialogue for the cutscenes was clunky and hard to understand: Disagree

The story elements (NPCs and cutscenes) were consistent in their world building, i.e. didn't contradict themself within the world the game has created: Strongly Agree

The story elements (NPCs and cutscenes) consistently added to the horror theme of the game: Neither Agree or Disagree

Evaluation of Results

Common Issues & General Options for improvement:

• NPCs didn't add to the horror themes of the game.
• Controls to interact with NPCs were generally well received, main issue was pressing E and then SPACE to progress the dialogue.
• Tutorial NPC seemed to clean for a post apocolypic world.
• Pilot NPC had no perceived relevence to the game.
• Dead body NPC was too hard to notice or missed completely.
• Injured Firefly NPCs line were perceived to be comedic rather, which took away from the horror themes of the game.
• There are no instructions on how to use bandaids to heal.
• Not sure why the player needs to be referred to as a "man", would prefer gender neutral terms.
• Where the Safehouse is situated was unclear for the level 1 to level 2 cutscene.
• No emotional connections to any of the characters.
• Cutscene could include the player reflecting about their situation to both create emotional connection and add to the horror themes.

Currently Actionable Improvements:

• Allow the key E to progress NPC dialogue as well as the key SPACE.
• Rewrite all NPC dialogue to be more inline with the horror theme and realistic within the apocolypic setting. The new script won't contain comedic elements or gendered terms.
• Add emotional hooks to NPC dialogue so that the user cares more about the NPCs.
• Modify the tutorial guys portrait so that he looks more rough/ragged.
• Have the pilot mention about how to use bandaids to heal.
• Make the pilot a recurring character to create an emotional connection with the user.
• Move the dead body to a safe area in level 2.
• Add player introspection dialogue to the in-between cutscenes.