A B Testing Protocol for Focus Groups - RutgersGRID/DanceAutism GitHub Wiki

BrainDance Game - A/B Testing Protocol for Focus Groups

1. Testing Overview

1.1 Purpose

This protocol outlines procedures for conducting A/B testing with focus groups to evaluate key design elements of the BrainDance Game. The testing aims to gather data on participant preferences and effectiveness of different design approaches before finalizing the vertical slice implementation.

1.2 Testing Structure

Individual sessions: 30-45 minutes per participant
Small group sessions: 60-90 minutes per group (3-5 participants)
Mixed methodology: Direct observation, guided feedback, preference selection

1.3 Participant Demographics

Primary: Children and young adults on the autism spectrum (ages 8-16)
Secondary: Parents, caregivers, and therapy professionals
Mix of genders, mobility levels, and support needs
Prior experience with technology/games noted but not required

2. Visual Art Style Testing

2.1 Test Elements

Option A vs. Option B comparisons for:

Character/avatar design
Environment/background style
UI elements (buttons, menus, feedback indicators)
Color palette options
Visual feedback effects

2.2 Testing Methodology

Side-by-side comparison
- Present both options simultaneously on split screen
- Allow time for participants to observe both versions
- Record initial reactions and time spent looking at each option
Engagement measurement
- Track eye movement/focus between options (if equipment available)
- Note which elements draw attention first and longest
- Observe physical reactions (leaning forward, pointing, etc.)
Guided questions
- "Which one do you like better?"
- "Which one is easier to understand?"
- "Which one would you want to play with again?"
- "Is there anything confusing about either option?"

2.3 Data Collection

Preference tallies for each option
Qualitative notes on reactions
Quotes from participants
Sensory response observations
Parent/caregiver observations

3. Gameplay Testing

3.1 Test Elements

Option A vs. Option B comparisons for:

Movement recognition sensitivity levels
Feedback timing (immediate vs. delayed)
Instruction delivery methods
Difficulty progression
Target/objective visualization
Point system mechanics

3.2 Testing Methodology

Sequential testing
- Participants try Option A followed by Option B
- Alternate starting option between participants to control for order effects
- Allow for rest periods between tests
Performance metrics
- Success rate with each system
- Time to complete equivalent tasks
- Number of attempts needed
- Signs of frustration or enjoyment
- Sustained attention duration
Facilitated feedback
- "Which one was easier to play?"
- "Which one helped you understand the movements better?"
- "Which one made you feel good about your movements?"
- "Which one would you want to play again?"

3.3 Data Collection

Completion rates for each version
Time-on-task measurements
Observed emotional responses
Verbalized preferences
Physical engagement level

4. Animation Testing

4.1 Test Elements

Option A vs. Option B comparisons for:

Movement demonstration style
Transition animations
Feedback animations
Celebration/reward animations
Character movement fluidity

4.2 Testing Methodology

Observation-based assessment
- Show animation sequences from both options
- Observe attention levels and emotional responses
- Note mimicking behavior or attempts to follow along
Comprehension testing
- After viewing animations, ask participants to demonstrate the movement
- Assess accuracy of movement reproduction
- Note which details were retained or missed
Preference dialogue
- "Which one was easier to follow?"
- "Which movements looked more like how you would move?"
- "Which one was more fun to watch?"
- "Was either one too fast or too slow?"

4.3 Data Collection

Movement reproduction accuracy
Attention span measurements
Emotional response indicators
Expressed preferences
Sensory reaction observations

5. Music Testing

5.1 Test Elements

Option A vs. Option B comparisons for:

Tempo and rhythm options
Instrumentation choices
Dynamic vs. consistent audio
Audio feedback sounds
Voice guidance vs. music-only options

5.2 Testing Methodology

Listening sessions
- Play audio samples without visual elements
- Observe physical responses (movement, rhythm following)
- Note emotional reactions and engagement
Movement synchronization
- Test how participants naturally move to different audio options
- Observe which audio elements trigger movement responses
- Assessment of rhythm following abilities with each option
Integrated testing
- Test audio options with visual elements
- Compare engagement with audio-visual combinations
- Note preferences when elements are combined

5.3 Data Collection

Movement-music synchronization success
Emotional response to different audio elements
Duration of engagement with each option
Physical responses (foot tapping, head nodding, etc.)
Verbalized preferences

6. Testing Session Structure

6.1 Individual Session Flow

Welcome and acclimation (5 min)
- Introduction to testing environment
- Explanation of process (appropriate to participant's understanding)
- Comfort establishment
Sequential element testing (20-30 min)
- Visual style comparisons
- Gameplay testing
- Animation comparisons
- Music testing
Closing feedback (5-10 min)
- Overall preference questions
- Open-ended feedback opportunity
- Thank you and next steps

6.2 Group Session Flow

Welcome and introduction (10 min)
- Overview of testing process
- Ground rules for group participation
- Ice-breaker activity
Rotation-based testing (40-60 min)
- Small groups rotate through testing stations
- Each station focuses on one element (visual, gameplay, animation, music)
- Facilitator at each station to guide process
Group discussion (15-20 min)
- Facilitated group conversation
- Voting on preferences
- Collaborative feedback

7. Adaptation Considerations

7.1 Communication Adaptations

Visual choice cards for non-verbal participants
Simple language options for all questions
AAC device accommodation
Interpreter availability if needed

7.2 Sensory Adaptations

Quiet space availability
Lighting adjustments
Volume control options
Fidget tools available
Break options clearly communicated

7.3 Mobility Adaptations

Accessible testing environment
Alternative input methods
Adapted movement expectations
Rest periods between activities

8. Analysis Framework

8.1 Quantitative Measures

Preference percentages for each option
Performance metrics comparison
Engagement duration analysis
Success rate comparison

8.2 Qualitative Analysis

Thematic coding of responses
Behavioral observation patterns
Emotional response categorization
Facilitator insights

8.3 Decision Making Process

Criteria for selecting winning options
Weighting system for different feedback sources
Process for handling mixed results
Implementation planning based on results

9. Documentation Templates

9.1 Observer Recording Sheet

[Template for documenting real-time observations during testing]

9.2 Participant Feedback Form

[Age-appropriate feedback collection templates]

9.3 Data Aggregation Framework

[Structure for compiling and analyzing results]

10. Schedule and Resources

10.1 Testing Timeline

[Specific dates and times for testing sessions]

10.2 Required Resources

Testing facility requirements
Technical equipment needs
Staffing requirements
Participant recruitment plan

10.3 Budget Allocation

[Budget breakdown for testing implementation]

Notes for Implementation:

Ensure all testing materials are prepared in advance with backup options
Prioritize participant comfort and engagement throughout
Be prepared to adapt testing protocols based on individual needs
Document all feedback systematically for thorough analysis