Game Testing - GregLinthicum/From-Logistic-Regression-to-Long-short-term-memory-RNN GitHub Wiki

Game testing tools

AI research

Further selected readings:

Overwatch

Overwatch - coaching

NeosVR

NEOS for Virtual Reality (VR) refers to NeosVR , a versatile virtual reality metaverse platform designed for creating, sharing, and exploring 3D and VR environments. NeosVR acts as its own testing platform by providing real-time editing, collaboration, debugging, and isolated environments for testing VR experiences.

  • NeosVR allows users to build and modify environments in real time while fully immersed in VR.
  • NeosVR’s LogiX system, a visual node-based scripting tool, allows creators to program interactivity and behavior directly in VR.
  • Unity Test Framework (Unity) is a great way to test any Unity content before importing it into NeosVR.
  • SteamVR Performance Test can help you gauge the performance for various headsets and setups.
  • VRAT (Virtual Reality Application Testing Framework) - can be adapted.
  • TestCafe (WebVR Testing)
  • XR Test (by Unity) (Extended Reality, including VR and AR).
  • PerfHUD (NVIDIA). PerfHUD is a powerful performance analysis tool, used to profile VR applications and spot performance bottlenecks at the hardware level (especially useful for high-end VR experiences).
  • VRTK is a popular toolkit for creating and testing VR interactions, especially within Unity. It includes tools for managing hand interactions, object grabbing, and movement mechanics, which can be tested before being integrated into NeosVR.
  • If NeosVR is connected to a backend through APIs (e.g., RESTful APIs), Behat could be used to test the behavior of those APIs. Behat is a PHP-based testing framework used for behavior-driven development (BDD).

KPI in Game Testing

In the context of video game testing, KPI stands for Key Performance Indicators. These are measurable values that game developers and testers use to evaluate the performance, quality, and success of various aspects of a video game. KPIs help track progress, identify issues, and ensure the game meets desired objectives before and after release.

Common KPIs in Video Game Testing:

1. Performance KPIs:

  • Frame Rate (FPS): The number of frames per second the game renders. A high, consistent frame rate ensures smooth gameplay. Frame rate drops or stuttering can indicate performance issues.
  • Loading Times: Measures how long it takes for different game levels or assets to load. Long loading times can negatively affect the player experience.
  • Memory Usage: The amount of RAM the game consumes during play. High memory usage could lead to crashes or poor performance, especially on lower-end systems.
  • CPU & GPU Usage: The percentage of the processor and graphics card's resources being used. Overuse can indicate inefficient coding or the need for optimization.
  • Crash Rate: The frequency of game crashes. A low crash rate indicates stability.
  • Latency: For multiplayer games, the delay between player input and the game's response (commonly called "lag"). Low latency is crucial for smooth, responsive gameplay.

2. Quality KPIs:

  • Bug Count: The number of reported bugs, categorized by severity (e.g., critical, major, minor). The fewer bugs, the more polished the game.
  • Test Coverage: A measure of how much of the game’s code, mechanics, or features have been tested. Higher coverage ensures more reliable testing results.
  • Defect Rejection Rate: The percentage of bugs or issues reported by testers that are deemed non-valid or not reproducible by developers. A high rejection rate can indicate communication or testing issues.
  • User Satisfaction (via testing): Testers' feedback on gameplay experience, controls, graphics, and overall enjoyment. Positive feedback during testing can be an early indicator of player satisfaction after launch.

3. Gameplay KPIs:

  • Completion Rate: The percentage of testers who finish the game or specific levels. A low completion rate may indicate difficulty spikes, confusing objectives, or gameplay imbalance.
  • Player Retention: For games with extended playtesting periods, this measures how many testers return after the initial play session. Low retention may indicate boredom, frustration, or lack of engaging content.
  • Time-to-Completion: The average time it takes players to finish a level or the entire game. This KPI helps balance pacing and difficulty, ensuring players aren’t stuck too long or breezing through content too quickly.
  • User Actions and Patterns: Tracking in-game behavior such as how often players engage in side quests, use certain mechanics, or follow story paths. This data helps ensure features are being used as intended.
  • Daily Active Users (DAU): The number of unique users who play the game each day.
  • Monthly Active Users (MAU): The number of unique users who play the game each month.
  • Session Length: The average duration of a single play session.
  • Playtime: The total time players spend in the game across all sessions

4. Network/Multiplayer KPIs:

  • Ping/Network Latency: Measures the delay between a player's action and the server’s response in multiplayer games. Lower ping equals a smoother online experience.
  • Matchmaking Efficiency: How quickly players are matched with opponents or teammates, and how balanced the matches are in terms of skill.
  • Server Uptime/Availability: The percentage of time game servers are available for players. Frequent downtime or crashes can negatively impact the player experience.
  • Packet Loss: The percentage of data packets that fail to reach their destination in a multiplayer game. High packet loss can cause erratic behavior or disconnections.

5. User Experience (UX) KPIs:

  • UI/UX Usability Issues: The number and type of issues players face while navigating menus, HUDs, or other interface elements.
  • Accessibility Testing: Tracking how well the game supports accessibility features (e.g., colorblind modes, controller remapping) and how easily testers with different needs can play the game.
  • Control Responsiveness: The time it takes for player input to translate into on-screen actions. Unresponsive controls can degrade the gameplay experience.

6. Monetization KPIs (for Free-to-Play games):

  • In-Game Purchase (Microtransactions) Success Rate: The percentage of players who successfully make purchases. Bugs in the purchase system can lead to lost revenue.
  • Engagement with Paid Features: The number of players who interact with or buy additional features like cosmetic items, level packs, or currency.
  • Ad Performance: For games that rely on in-game ads, tracking ad impressions, click-through rates, and ad revenue is important.
  • Cost Per Install (CPI): The cost associated with acquiring a new user who installs the game.
  • Return on Ad Spend (ROAS): The revenue generated from advertising compared to the cost of the ads.
  • Average Revenue Per User (ARPU): The average revenue generated per user.
  • Average Revenue Per Paying User (ARPPU)
  • Lifetime Value (LTV)