VDK Logs Usability Testing - vmware/versatile-data-kit GitHub Wiki

Table of Contents

Purpose

The purpose of this usability testing is to evaluate the effectiveness and clarity of our logging system. By engaging users in a hands-on experience with a pre-configured environment, we aim to observe how they interact with the logs to identify and resolve errors.

Background

Objective:

The primary goal is to assess how users interact with VDK, especially in terms of navigating logs and identifying as well as troubleshooting errors.

Participants:

  • TO BE ADDED

Environment:

We will use this pre-configured environment for the usability testing:

Binder

The environment comes with a ready failing job for the participants to work with.

Data Job for Testing:

Job Design:

Data processing job within VDK that is intentionally designed to fail due to a common and identifiable error. The job is simple yet reflective of real-world tasks users might encounter.

Task for Participants:

  1. Run the Job: Instruct participants to execute the data job.
  2. Assess the Logs: Participants should closely examine the logs generated during the job execution to gain insights into the job's behavior and error specifics.
  3. Identify the Error: Participants should use their assessment of the logs to identify the specific cause of the job failure.
  4. Attempt to Fix: After identifying the error, participants are tasked with implementing a solution based on their understanding and available resources.
  5. Re-run the Job: Once the fix is applied, participants should re-run the job to verify if their solution successfully resolved the issue.

Usability Testing Process:

  • Introduction: Briefly introduce participants to the testing environment and the task at hand.
  • Execution: Allow participants to run the job, examine the logs, and attempt to rectify the error.
  • Observation: Observe and record their approach, focusing on how they interact with the logs and the troubleshooting steps they take.
  • Feedback: Conduct a short post-task conversation to gather feedback on their experience, challenges faced, and suggestions for improvement.

Post-Testing:

  • Report on Data Collected: Analyze common patterns, difficulties, and areas of improvement identified during the testing.
  • Action Items for Improvements: Develop a list of actionable steps to enhance the usability and effectiveness of the logging system.