User Evaluation - 2024-CMPU9010-GROUP-3/magpie GitHub Wiki

1. Define Objectives

Goal: To assess the usability, functionality, and overall experience of the map-based application. Key Questions:

Primary Users: Working professionals of Dublin looking for nearby amenities.
Secondary Users: Locals, tourists, students, and commuters in Dublin.
User Characteristics: Age range (e.g., 18-65), diverse levels of tech-savviness, familiarity with map apps, both locals and new visitors.

Prepare scenarios to simulate real-world use. Focus on tasks that align with the app’s core features:

Basic Search: Ask users to search for a common amenity (e.g., "nearest pharmacy").
Filter and Sort: Test filtering by categories (e.g., cafes, parks, ATMs) and sorting options.
Navigation and Directions: Have users select an amenity and view directions to it.
Review and Ratings: Include any functionality for viewing or submitting reviews, allowing users to explore and add feedback.
Save or Bookmark Amenities: For any feature that saves locations for quick access.
User Feedback and Reporting Issues: Test feedback features to report errors in locations or add new amenities.

Usability Testing: Remote or In-person Testing: Set up controlled sessions, asking users to complete predefined tasks while thinking aloud. Record their screen, facial expressions, and voice to capture reactions.
Contextual Inquiry: Follow users as they test the app in real locations (e.g., finding nearby amenities when out in Dublin) to observe real-time behavior and usage patterns.
Surveys and Questionnaires: Collect quantitative data on user satisfaction with a post-test survey. Include Likert scale questions (e.g., ease of use, visual appeal, accuracy of results) and open-ended questions to gather qualitative insights.

Quantitative Metrics:

Task Success Rate: Percentage of users who can complete each task.
Time on Task: How long it takes to complete each task.
Error Rate: Frequency and types of errors encountered.
System Usability Scale (SUS): A standard 10-question scale to measure usability.

Qualitative Metrics:

User Satisfaction: Ratings from post-task surveys or interviews.
Observational Notes: Note users' facial expressions, comments, and hesitation points.
User Comments: Gather specific feedback on features, aesthetics, and pain points.

Pre-Test Briefing: Explain the purpose, assure confidentiality, and encourage honesty.
Introduction to the App: Provide a brief explanation but avoid instructions on navigation to gauge intuitiveness.
Conducting the Tasks: Observe users without interfering. Prompt them to “think aloud” as they navigate.
Post-Test Interview: Ask follow-up questions on ease of use, design appeal, clarity of directions, and perceived accuracy.
Thank Users and Offer Incentives: Consider small incentives, like vouchers or gift cards, to encourage participation.

Compile quantitative data to identify clear success metrics.
Analyze qualitative data to identify recurring themes in user comments and observational notes.
Use affinity mapping or thematic analysis to cluster feedback on similar usability issues or positive aspects.

Prioritize Findings: Focus on fixing high-impact issues, like navigation issues, inaccurate results, or confusing UI elements.
Re-test: Conduct follow-up testing on key problem areas to verify improvements.
Continuous Feedback: Incorporate a feedback loop in the app, allowing users to report issues and suggestions even after launch.

Summarize Findings: Create a report highlighting key usability issues, positive feedback, and actionable insights.
Present Recommendations: Suggest design, functionality, or feature improvements based on the data.
Roadmap for Iterations: Outline the timeline and resources for implementing changes based on priority and feasibility.