Methodology

Overview

User testing will be performed on the base item mechanics of the game (including ammo and reload) in order to ensure the implementation makes sense and fits into the pre-existing world. Therefore this will include, picking up items/ammo, keeping track of & using items/ammo and reloading (including relevant keymapping). Additionally, the user abilities will also be included in the testing as it is relevant to the implementation of items (as it will later be implemented as a pick up able), though this will more be in the context of the usage and options than the current way to pick things up (with further testing implemented for the ability to switch through pickup-ables when implemented in later sprints). In order to achieve this potential users will be given access to the game and requested to undergo a series of tasks to test their inherent understanding and get opinions on certain mechanics. The testing will run first as a short thinking aloud exercise and request users to explain their decisions when undergoing a task to get better insight into their assumptions (to test the basic intuitiveness of the mechanics) alongside a short Q&A to better understand their opinions on the items (usefulness, appropriateness, etc)

Justification

Items were chosen to be tested as it is a new addition to the game and therefore it is important to get the perspective of a potential user to see whether the components make sense both as a system and in its presentation within the game world. Without testing, any issues would only be found much later and any future additions/progress could be wasted time as the system could need to be rebuilt to ensure player experience is prioritised (as the purchasing/item system is an intended feature for the final game).

Testing Process

User Brief

Before testing the user will be given a brief description of the game and basic descriptions of the player functionality (i.e. the player is capable of attacking and getting hurt) alongside the non-item related controls (i.e. they will not be told about the hotkeys, shooting and reloading buttons). The test administrator will explain how the player is to tell them all their thoughts while completing the given tasks, without asking for guidance during play related to the items (except for if the user becomes entirely stuck -> important to not let the user try for too long if they appear uncomfortable or frustrated/make sure they know they can simply say they have no idea to ensure user is comfortable during testing. Additionally make sure in phrasing to say that the user isn't expected to get this perfect when presenting tasks related to key bindings so they do not get as frustrated).

Note: As certain functionality is not currently present for abilities you will need to switch between to have the player attempt them and more explanation will be needed to explain what is happening (especially for invincibility which as of yet has no visual information). However, users will still not be told what the ability button is just that they have an ability (except for if the user becomes completely stuck -> same methodology as above).

User Tasks

The activities the user will be asked to perform will mostly include both basic concepts (e.g. how to pickup items).

1. Pickup all the items on the screen (test whether they understand how to pickup items and what is an item)
2. Use the bandage Item (test their intuitive understanding of how to use items)
3. Use the rock (test their understanding of the keymapping and see if they understand that the ammo is different to the other items)
4. Reload your ammo (test the mapping of reloading)
5. Use your ability (first for the dash, then again once switched)

Small Questionnaire

1. How did you feel about the keymapping? (hotkeys, reloading, shooting) Would you prefer different ones?
2. Outside the inclusion of a tutorial would anything else make it easier to understand the controls?
3. Did you feel that the items would be useful?
4. Did anything seem redundant/unnecessary?
5. How did you feel about the abilities?
6. Did you feel any information was missing (item/abilities)?
7. How would you feel if abilities were switched by walking over an item that changed to another one?
8. What did you think of the abilities? Were they useful? How did you feel that you could only use one?

Testing

Participant #1: Significant observations and feedback

Observations and Feedback:

• User understood that you just had to walk over items to pick them up
• User understood what items could be picked up
• User figured out items were used by numbers (how do you use multiple items with different buttons?)
• User said if numbers didn't work they would have tried arrows (said arrows could be better for few items)
• Took the user a while to use the bandage
• User understood the ammo was ammo and related to ranged attack and not item hotkeys
• User saw the reload but took a while (tried to reload before out of ammo)

Questionnaire:

• Enter is okay for shooting, didn't feel random (wouldn't prefer mouse directly -> feels more keyboard)
• Would like a tutorial (especially the hotkeys and reload)
• Felt the bandages are useful
• Didn't know why there were coins but could see them being used elsewhere in the game
• Didn't figure out the ability key and didn't have a key they would prefer
• Wanted indication for ability cooldown, didn't understand why it couldn't be used again
• Though long dash was a useful ability but felt like they would use others instead in later game
• Liked the invincibility ability
• Prefer if the changing abilities was an activation (couldn't accidently change abilities)
• Made sense that you could only use one at a time

Participant #2: Significant observations and feedback

General Feedback based on questions asked

• Made sense that items can be picked up by walking to them
• Game feels quite complete but would be nice to have attacking abilities (like having a bar that charges up the more you hit enemy with melee or range and being able to use an explosion ability that damages enemies close by)
• Would be nice to have more clothes and items to buy in game

Participant #3: Significant observations and feedback

Observations and Feedback:

• User understood that you just had to walk over items to pick them up
• User understood what items could be picked up and didn't try to pick up bushes/cobwebs
• Took the user a while to try 1 for the bandage (tried multiple letter and control keys first)
• Felt that they would have tried 1 first if there were more items / the item HUD had them in boxes with numbers
• First tried the hotkey for using the rock, then mouse, then Q, then the control keys (used enter then)
• Said enter was fine but not a first choice with no knowledge on the controls (and not knowing it was a keyboard only game)
• Saw the reload message easily and understood to use R (said would have tried R first if they hadn't seen the message too)
• User could not figure out how to trigger the ability on their own (tried a couple then said they had no idea)
• User said that/tab/caps lock)

Questionnaire:

• (already described all issues with key-mapping and preference on changes)
• Felt like a tutorial would help a lot but also would have liked a way to see the controls after the tutorial
• Thought all items/abilities seemed useful & not redundant
• Felt that there could be some indication for when the ability cooldown is over but overall felt they had most of the information (outside key bindings and all the exact things a player is capable of)
• Understood the choice to not have multiple abilities at once (felt it would be tactical/more engaging)
• Felt the invincibility would be too overpowered (needed to give it a shorter duration/not allow attacking during)
• Felt the dash wasn't the most useful currently but would be useful for enemies with AOE attacks or if more enemies were present on screen at a given time
• Liked the user of a long cooldown for abilities
• Liked the idea of switching abilities by walking over them like items (don't like that you currently are unable to switch)

Participant #4: Significant observations and feedback

Observations and Feedback:

• Identified movement (WASD) controls successfully
• Tried using mouse to aim and click to fire range attack
• Understood that player had to simply walk over items to pick them up
• Didn't find hotkeys to use items, had to be told of keybind
• Attempted to use items / attack within safehouse to 'trial' player capabilities
• Picked up all items successfully, only got hurt 2 times
• Felt that items should have a modifier / different stats
• Only discovered ability (Left Shift) when told by test administrator

Questionnaire:

• When prompted with hotkeys, agrees that keybinds are okay
• Prefers attack key something other than Enter
• Prefers J for some ability, due to nub on most keyboards
• After playing through game, health system isn't clear; "when are bandages supposed to be used?"

Participant #5: Significant observations and feedback

Observations and Feedback:

• First attempted to use arrow keys to move
• Found melee attack (spacebar), then found movement (WASD) easily.
• Trying all keys, accidentally triggered bandage.
• Didn't find Ranged Attack, had to be told by test administrator
  • "Why would it be Enter?"
• Correctly Understood player had to walk over items to collect them.
• Unsuccessfully tried many times to aim at enemy while retreating (both melee and ranged attack)
  • "Aiming of sword is terrible"
  • Shooting rock in specific direction is not consistent
• Hit the safehouse multiple times, hit box is too large
• Can't dodge bullets, really clustered generation through most games
• "Dash ability is useless"

Questionnaire:

• "All keybinds are awful, they're so arbitrary"
• Worst one being abilities (Left Shift), did not find it at all
• Agreed that tutorial would help, but preferred keybinds similar to other games
• Did not understand the purpose of items, or how they factored into the game
• "Is there a invincibility effect when dashing" - questions if dash ability either makes you invincible / hurts nearby enemies when running through them.

Participant #6/#7: Significant observations and feedback

• Horrible keybind, unintuitive, "keybinds are arbitrary"
  • eg. space for sword attack
  • "consider other keybinds similar to popular arcade or FPS games"
• user sees potential for bandages, just not in current state
• dash ability limited in current state, can easily run away without dash
• possibly projectile control with mouse
• user sees potential for special items to change player ability

Participant #8: Significant observations and feedback (ran out of time)

• When prompted on player capabilities, user found all keybinds through trial and error.
  • Agrees that most keybinds make sense
• Activated Sticky Keys with dash ability multiple times.
• Agrees that having items that change the player's ability makes sense, although there has to be an indicator similar to items on which ability is selected

Participant #9: Significant observations and feedback

• Identified movement (WASD) and melee attack (spacebar) quickly.

• Required prompt from test administrator for ranged attack, aiming, and dash ability.

• Successfully picked up all items. Thought cobwebs and bushes were a special type of item.

• Used trees to hide from enemies

• Agreed that some keybinds make sense

  • Ranged attack (Enter) should be replaced
  • Mouse to aim?
  • Left shift is bad, sticky keys. Maybe move to F (neighboring WASD keys)

• "Abilities make sense in the grand scheme of things, but the levels have to be more complicated or bigger"

• Agrees that changing abilities is a interesting idea and brings different ways to defeat enemies

Evaluation of Results

Common Issues & General Options for improvement:

Most users had issues with the ability key binding (alongside the key binding of many other functions of the player: the ranged attack and item use).

• Alter the HUD to indicate that items use the hotkeys
• Change the mapping for these functions
• Incorporate the mouse for interaction (specifically for ranged attack)
• Include a tutorial to show the keys directly and not ask user to guess them during play (important)
• Include a key mapping page in the settings (either a static explainer or a customizable settings page)

Most users had a problem with understanding when the ability could be used (no cooldown indication _. could be single use or multi-use).

• Including a cooldown indication on the screen
• Having a message/animation appear when the player can use their ability again

A user mentioned that the invincibility ability seemed to overpowered in the current state of the game.

• Remove the ability to attack during the invincibility
• Make the invincibility shorter
• Make the invincibility an ability only available in later game levels

Currently Actionable Improvements:

• Make the character unable to attack during invincibility

Other improvements involve more discussion and larger programming changes which will be considered within the next sprint to ensure all changes are done in the best way.

Evaluation of Methods

Overall the testing was generally successful, however due to a lack of feedback and function to switch abilities in game the coverage for testing the abilities was quite small. Therefore, these elements would need to be retested in the future to ensure they fit well (especially in the context of combat which was not a focus in this testing document). Additionally, the method for testing feedback was not very insightful as most users simply are unable to think of the correct key but are unsure what key binding would be best due to the very broad nature of the question. In the future different testing methods will need to be researched and implemented to better optimise key bindings for the general user (alongside testing following the use of a tutorial as will be the case in the final game).