Methodology

Overview

In finalizing the game it is important that the players experience the main interactions of the game (combat & using items) at the intended level of difficulty and therefore have the planned experience. In order to ensure this, user testing must be performed to generally balance the items and the general combat of the game. This includes the damage and range of each weapon available to the user, as well as the player's abilities and bandages. Additionally, this will also act as a way to get feedback on the first level of the game to ensure the enemy arrangement & difficulty is correct for a tutorial/introduction level.

In order to complete this users will perform a form of play testing where they are given access to the game and asked to complete levels under different given sets of weapons and abilities. The tester will not guide the player other than telling them to complete the level and will take observations during play alongside a final questionnaire.

Justification

Overall play testing is the best way to get feedback on gameplay as it gives the player the most freedom and therefore would give the most accurate information on how they experience the levels with the given item sets. Additionally by focusing more on observations than asking the user to talk as they play allows the player to experience the game more on their own terms without distraction. In addition to that, specific notes were provided to testers to take note of (this is to ensure data would be less noisy that will be used to reiterate the game and balance anything in the game).

Due to a large number of options for weapons alongside the multiple abilities, complete testing cannot be done on every scenario and therefore the user will only be asked to complete the game under two different weapon sets (this also to ensures their change in skill level as they play the game does not heavily affect the results for items tested later). To accomplish this, the player will be tasked with completing the first 2 levels of the game - allowing the player to get different weapons/abilities and armour in between levels (to test the secondary set of weapons and abilities).

Testing Process

User Brief

The player will be given the general concept of the game, so they have an idea of the game scenario and the emotional motive of the player character. Note that during the game if the player asks the controls again the test coordinator will be able to tell them but should take note of this.

User Tasks

The player will be asked to complete level 1 (in order to get feedback on its layout and because it is the first level so it will be the most appropriate starting point for a new player). They will be asked to fight their way through the level however, users will not be required to kill all enemies but some of them (to somewhat test the balancing of weapons/abilities player starts with), simply following the lights and killing enemies in range will be enough.

Upon completing level 1, the player will be required to purchase (testers should provide a brief instruction about this - let the user know to attempt to make purchases - whether it may be weapons/abilities/armour) a different weapon and ability (or even armour if they have enough coins collected from the first level). Testers will be required to take note of the kind of weapons/abilities purchased and observe whether the weapons selected would require more balancing or if they are sufficient as they are now (this is because testing all combinations will be unachievable in the given amount of time, so testing logical combinations players themselves choose will give us insight without requiring too much time debt).

User Starting Items/Weapons/Abilities and Notes tester should be aware of:

• Bat & Long Dash (default/starter -> should be the most difficult)
• Defence level will be 0 (no armour - observe how the user does without any help)
• 3 bandages will be given (observe the effect of bandages - how much does it help the player to survive and does the player remember to use bandages often?)
• 10 bullets provided (does the player have a sufficient amount of ammo to shoot rocks?)
• 100 coins (this is to allow users to purchase a different weapon/ability or even armour if they wish to do so when advancing to the next level) Note. Comments in brackets are observation notes that can/cannot be taken while observing the player's behaviour while user testing is being completed.

Metrics measured during:

• Observations were taken: How the player fights, if they struggle, what they struggle on, if they seem frustrated/relaxed/bored, etc.
• Time measurements: Objective measurements on how long it takes them to complete the level (does not need to be strict - can be approximated)

Reflection questionnaire:

• Rate the difficulty of level out of 5 (level 1 and level 2)
• What do you think of the enemies that you have faced while exploring and completing the first level?
• Does the weapon you have equipped sufficient in terms of destroying enemies and exploring the first level? (could also comment about weapons used in level 2 as well)
• In terms of experience - what do you feel about the gameplay of level 1 and 2? Should there be more enemies? Did a difference in weapons enhance your gameplay experience? (or even ability/armour when purchasing them from the shop in safehouse)
• Overall, was the game fun? Is it difficult enough? (Provide a rating out of 5)
• Rating of game? (1 to 5) This provides a general overall critic of the game currently and what else needs to be focused on (tester ensure to ask user for a brief explanation about why this rating was given and if there is anything that needs to be better balanced or focused on)

Note: these questions all have follow up questions but should be asked each section at a time to not overwhelm the user

Note to testers

• Be aware of type of weapons used in level 2 and observe how user's experience have changed with different type of weapons used (this applies to if user has armor and when different ability is used as well)

Testing

Participant #1: Significant feedback

• Difficulty for level 1 - exploration was difficult because there was no mini map that clearly indicates where user is going during the game, user had to memorize while playing the game and was feeling excited/nervous when torch was running out. Difficult to manage resource properly when trying to learn the game. Bullets were not unlimited and during exploration of level 1, bullets picked up were not sufficient (user used ranged attack very regularly). Overall - it was probably 4/5 in terms of difficult (due to how it was difficult to get through level due to lack of mini map)
• Enemies sufficed since the numbers weren't as much as it is in level 2. It sufficed in terms of allowing user to learn the game and getting the basics down when trying to utilize abilities, ranged attack and melee attack
• For level 1 (already answered previously), in level 2 - user had crowbar and invisibility skill equipped (along with full set of armors and additional fuel - user died on level 2 and torch ran out). Level 2 was a lot more difficult - 1) because user realized the following: without health indicators for each enemy, it was difficult to tell if enemies were actually receiving damage and if it was worth the risk to receive damage and get item drops from killing enemy, 2) ran out resource (rocks) to fire range attack and too afraid to melee attack enemies (there were many enemies surrounding player during level 2) and 3) melee attack does not deal knockback to enemies (does not provide comfort zone for player to run away - so better to run away rather than even use melee attack). User described overall experience as the lack of feel of the game at times when trying to combat enemies in game.
• Depends on motivation or goal of the game. User's play style is to be able to be strong and have full control over game but for this game, it was difficult in a sense that user has to manoeuvre enemies in game rather than fight and get stronger as user completes the game. User provided 2 ratings, first for if game is to make it difficult for player to get stronger and have to try to survive by dodging enemies in an attempt to complete the game - user gave 5/5 but if user wanted to have full control and be the strongest that he/she can be to completely explore the game - a rating of 2/5 was given because of a number of frustration point highlighted previously.
• Overall, game was fun (4/5) but there are still a few things that needs to be balanced or buffed. Examples were - there should be a prompt provided to user when wanted to leave safehouse or even talk to shop keeper, the health regen does not come early enough to help user survive and recover and ability cooldown was not short enough for player to use as often as he/she would like to
• Note that user used crowbar as melee weapon, invincibility ability, bought additional fuels and was fully equipped when playing level 2. This is worth noting since user highlighted that it was still difficult even when player is at the highest in terms of defense level and had the ability to be invincible during game as well. User also added that it was difficult to shoot range weapon while walking away (would like to be able to shoot enemy while running away together).
• Note: user used ability often, never stopped to regenerate

Participant #2: Significant feedback

• Enjoyed exploring level 1 while killing enemies placed in same game level. Was able to fully explore map while engaging with all enemies without any hassle - rated difficulty of level 1 to be 1/5 (quite easy progress but sometimes wonder where to go while completing level). For level 2, a rating of 5/5 was given. This was focused specifically on how there were many enemies encountered (mobs of enemy surrounded player many times and there was no way of actually pushing NPCs back after using fire cracker (ability) or short dash. Player was excited to be able to encounter so many enemies at once but felt it was a bit too difficult to the point where it was frustrating
• Level 1 was balanced in the sense where it allowed player to explore and at the same time there was sufficient amount of enemies placed for player to interact and target them.
• Weapon and ability was sufficient but at times there were too many NPCs coming towards player and a better way to prevent enemy NPCs from swarming player so quickly is to add a layer of cushion or pushback from enemy to player to allow player to retreat and recover. Also added that fire cracker enabled player to run and recover but ability cooldown bar took too long to fill and with so many NPCs - it was difficult for player to run and dash away
• Overall, user commented that game is definitely fun and rated it with a 4/5 but commented that weapons were okay but perhaps NPCs placement should be better optimized to not overwhelm player so quickly in level 2. Having NPCs (at least as much as there is in level 1) is good but having too much may be too difficult - alternative is to either reduce number of NPCs in level 2 or buff player's melee (knockback) to allow player to recover or if there is a way to help player to fill ability bar quicker when attacking to incentivize to risk and attack NPCs (rather than gold since player can only use gold in safehouse which doesn't really provide an immediate incentive to player in game and doesn't justify the reason to risk getting attacked by enemy in game)
• Note that user never stopped to regenerate but used their abilities often in both levels

Participant #3: Significant feedback

• For level one they didn't struggle as all and explored the level (took approximately 5 minutes)
• Were confused when first attacking the enemy, thought it wasn't doing damage as there wasn't feedback (said to the tester it must be broken)
• Didn't realise at first that they had a melee weapon, when they asked for controls the tester told them they had a melee weapon and they were surprised they had one
• User relied on majorly the ranged attack for most enemies, only using melee attack for the small enemy (and occasionally the larger enemy)
• Often spammed the ranged attack to kill enemies, didn't move much during combat
• Bought the crowbar, full armour and a torch for the second level
• For level two they spent more time having to avoid enemies however they still didn't struggle. They also seemed to hit more dead ends while trying to progress (took approximately 8 minutes)
• When they had armour they powered through combat much more often, was able to power their way through fights
• Never left a fight to regenerate (but also never really needed to)
• Never used their ability past the first use when testing it

1. Level 1 was 1/5, which they said they expected from a tutorial level and would not want it to be more difficult. Level 2 was 2/5, which they said was okay, but given the game was so short it should maybe be a little harder.
2. For the small enemy they felt they were perfect; should kill in one hit, be able to stomp on, For the smaller eye they felt like they died too quickly and should be harder to kill; they look much bigger and scary, should be stronger than the small enemies. For the 3-shot and spawner they had no comments and felt they were as expected combat wise; not too strong/weak. For the final large enemy they felt it was okay but could maybe have higher health.
3. Felt the bat was fine, worked well for the enemies on level one. However, also felt the projectile was a bit too overpowered, was able to kill even the hard enemies very quickly by standing in place and spamming, felt like this shouldn't be able to happen.
4. Expected more enemies on level two but was happy with one (was getting used to the game so expected empty space) -> Mentioned that story wise it was weird that the place you are fleeing from has less enemies. Didn't notice too much of a difference in the second level but bought the crowbar which was like the bat they start with so didn't expect a large difference. Purchased armour and noticed they were able to take more hits, felt a little too strong (said there should probably be more enemies if the player has full armour).
5. For fun they said is was a 3/5 while for difficulty it was a 2/5. Felt like there wasn't much of a challenge.
6. Said they would rate the game a 2.5/5: has potential but currently not difficult enough for them to really get into it. Also stated that it was very short and that made it a lot harder to get into the game. Said for what it was it was generally good but compared to many games they've played (and personal taste -> plays a lot of open world AAA titles) it lacked a bit in challenge and length.

Participant #4: Significant feedback

• For level one they didn't struggle, explored most of the level (6 minutes)
• Used very few items in the first level
• Used both melee and ranged attack pretty evenly -> ranged mostly for the harder enemies as they stayed aware more often
• Bought a crowbar, invincibility and a couple torches for the second level
• Used ability rarely after changing, not at all before
• For level two they spent more time having to avoid enemies as more were present. Struggled a little but didn't appear frustrated
• Used up most of their items in the second level
• Never walked away from combat to try and regenerate

1. Level 1 was 1/5, which they said made sense as they were getting used to playing the game. Level 2 was 2.5/5, as it was definitely a step up but not real difficulty (new how to play, now had to actually fight enemies in groups)
2. For crawler they said it was easy and one hit as they expected (only every messed up when they were facing the wrong way while getting used to the controls). For the eye they said it was a little harder than the crawler but not by a lot, just cause you had to dodge before you could hit. Said the spawner was good, a little bit harder just cause there was more enemies in one spot, but said that cause they were weaker they didn't feel overwhelmed 9mentioned they definitely need more feedback that it can be killed though). The 3 shot one they said was harder cause it shot more and unlike the one eye in the darkness you couldn't see the spikes which made it harder (said it wasn't too hard though just more challenging). For the large enemy they said it was the most engaging to fight as it was pretty fast and needed to keep distance while hitting it (said it was the hardest but not too hard as long as there's not too many at once)
3. Felt the bat was okay, didn't really notice any difference between the bat and crowbar.
4. Felt the gameplay was similar between, just more playing then learning in the second. Got the invincibility upgrade and said they liked it as it allowed them more space to go around things when they don't have a lot of items left, which they liked doing (liked that it gave them more freedom to pick how to play). Felt there was a good amount of enemies in both levels.
5. For fun a 3.7/5 (a feeling, really liked the story, felt like the areas showed the journey/story direction well) while for difficulty it was a 2.5/5 (some challenges but not super hard to overcome, didn't get frustrated or die a lot, but still used a lot of resources in second level).
6. Said they would rate the game a 3/5: was fun, cute looking, not too difficult but still had challenges, but was simplistic/short and did want to go back to shop but couldn't so it was missing some extra niceties.

Participant #5: Significant feedback

• Had trouble seeing the bandages on the ground, both in the safehouse and in the levels
• Didn't realise that the rocks on the ground could be picked up and were ammo
• Didn't use the ranged attack much, kept forgetting about it. Once they got used to the game, they used it a lot more and would spam tough enemies to kill them quickly
• Had trouble talking to NPCs (bug identified by Team 1)
• Had trouble getting through level 1 as they couldn't figure out which way to go. Suggested some kind of objective marker
• Didn't seem too concerned with the torch running out
• Died a few times on level 2 but this was due to bad performance on level 1 and still getting used to the game
• Thought there should have been more hints and tips in the game about things like health regen, darkness, torch fuel, bandages, etc.
• Liked how directed and difficult level 2 was, especially with the dead-ends with resources
• First choose armour, sledgehammer, firecracker, and a torch for level 2 but after dying a few times switched to helmet, armour and sledgehammer
• Didn't use the special abilities much, so much so that they opted for both armour pieces instead of changing their ability
• User never used their ability
• User did not stop to regenerate health, likely didn't realise they could

1. Level 1 was a 2/5, mainly just due to getting lost and used to the game but the actual enemy balance was a 1/5, good for a tutorial. Level 2 was more 4/5 for difficulty, not because it was particularly difficult but because they were still getting used to the game.
2. Felt the enemies were pretty well balanced and diverse. Liked the little enemies being one hit but maybe the large enemy could be a bit stronger. Didn't realise you could kill the spawners until I asked them about it.
3. Liked the bat at the start, felt it was a good balanced starting weapon. Much preferred the sledgehammer when they got it, they loved the long range and heavy damage on it.
4. The main issue they had with level 1 was navigation while level 2 they felt was fine. In fact, they preferred the structured and tight layout of level 2 with all the dead-ends, felt they knew better where they were going.
5. Pretty fun, they would rate it 4/5
6. They'd give it a 4/5 at the moment, felt that it actually looked like a game. Liked the graphics, feel, and atmosphere of the environment and that kind of stuff. Still, places to improve but it's looking good now.

Participant #6: Significant feedback

• The story is too long at the start and it is not clear how to control different abilities.
• They did not know where to go to exit level 1. They did not know the location of the safehouse.
• They focus on killing enemies rather than exiting the level
• They died from the attack of the large enemy in level 1. When they start from the last check point the game starts from the beginning, which is annoying.
• They did not know how to use bandages and fuel.

1. The difficulty for level 1 is 3. Because the large boss is strong and it is a little bit confusing to exit the level. the difficulty for level 2 is 4. Although it is harder to finish the level, when the user understands the game, it is not too hard to get through the level.
2. The enemies are great, but the large enemy is too hard to kill.
3. Yes the weapons are enough.
4. Generally, the experience was good and varied weapons enhance the experience. The number of enemies is suitable as well. The shop idea is good, but it is not clear how to use different items.
5. Generally, it is fun and it is difficult enough.
6. The game is rated as 4. The story is too long and the introduction to the gameplay is not clear. But the game is complete and sufficient as an assignment.

Evaluation of Comments

Common Significant Feedback

Had issues with users spamming the ranged attack from far away from the enemies and not using the melee attack which made fights far too easy leading to users getting less enjoyment out of the game as they are not being challenged.

• Reducing ranged attack damage
• Reducing available ammo in maps
• Reducing starting ammo
• Reducing range of ammo (so they have to get closer)
• Making melee damage stronger overall
• Increasing melee attack speeds

None of the users stepped back from fights to allow regeneration making the regeneration feature obsolete.

• Change tutorial dialogue to inform the player that they regenerate health
• Decrease health regeneration time so user notices regeneration/will see waiting to regenerate as viable
• Add animations to the HUD when regenerating to visually show regeneration

No user testers commented on the player making use of the abilities as overall users did not use the ability often after the first use.

• Make the ability cooldown lower
• Have a way to decrease the cooldown (e.g. killing enemies / reaching different sections in map / collecting enough coins)

Responses to Feedback

• Reduced ranged attack damage from 5 to 3 to reduce 'spamability' of the ranged attack
• Reduced starting ranged ammo from 15 to 10 to reduce reliance on ranged attack
• Increased player melee weapon knockback to make melee more viable
• Reduced wait time for health regen so players can more easily notice that they regenerate health
• Increased torch timer so that it doesn't run out so quickly (because during testing multiple testers noticed users running out of torches in most map runs, even when buying torches -> which was not intended)
• Automatically give the player one bandage and additional torch time on reaching the safehouse
• Made it so that melee weapon kills reduced ability cooldown by 10% (of total amount) to increase the viability of abilities
• Increased health of the large tough enemy to increase their threat

Evaluation of Methods

Overall we were able to get significant feedback about main issues with game design that led to the user playing the game in an unintended way which negatively effected their experience during the game (i.e. spamming ranged). Therefore, our testing was generally successful in achieving the goal we set out for in testing.

In retrospect, in order to test the combat in isolation it could have been beneficial to test on only one level and therefore be able to directly compare differences in the strength of weapons/ability/items. Alongside this having more users tested and possibly encouraging users to try out different weapon 'classes' would have been beneficial to get more rounded feedback and be able to get feedback on all weapon types. Therefore in the future, when testing, more control measures will be implemented alongside increasing the number of users in order to get more actionable feedback that is better representative of issues.