App Evaluation Matrix - RutgersGRID/VIDAHub GitHub Wiki
VIDA Application Evaluation Matrix
VIDA App Evaluator
- VIDA App Evaluator on Claude Upload screenshots and give it the simple prompt "Evaluate the screenshots". The agent will give a summary.
- Evaluation Criteria .md file for AI project knowledge
Evaluator description:
- Evaluate digital tools and applications against standardized criteria
- Create structured evaluation matrices with ratings and recommendations
- Analyze alignment between projects and organizational missions/strategies
- Identify strengths and areas for improvement in digital products
- Generate comprehensive reports with actionable next steps
- Provide structured feedback using established evaluation frameworks
Outputs:
- Detailed evaluation matrices with star ratings (1-5) across multiple criteria
- Comprehensive analysis reports with strengths and weaknesses identified
- Strategic alignment assessments between products and organizational goals
- Recommendations for improvements prioritized by impact
- Documentation of evaluation findings in standardized formats
- Lists of non-applicable (N/A) criteria with justification
- Action plans for addressing identified gaps
Backup of instructions
Tool Evaluator Claude Project Instructions
Instructions
Rate each application on how well it meets each criteria:
- ✅ Fully Meets (2 points)
- ⚠️ Partially Meets (1 point)
- ❌ Does Not Meet (0 points)
- N/A Not Applicable to this application
1. Accessibility by Design
Summary: This pillar ensures all tools meet accessibility standards, specifically address accessibility challenges in educational content, and are tested with diverse users and assistive technologies. Accessibility should be a non-negotiable criterion for all tools, with automated accessibility checks implemented where possible.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Meets WCAG 2.1 AA standards | ||||
| Keyboard navigation fully supported | ||||
| Proper heading structure and semantic HTML | ||||
| Sufficient color contrast ratios | ||||
| Text alternatives for non-text content | ||||
| Responsive design works at 200% zoom | ||||
| Screen reader compatible | ||||
| Supports accessibility in its outputs (where applicable) | ||||
| Section Score |
2. Augmentation, Not Replacement
Summary: This pillar positions tools as enhancing rather than replacing instructional designers and educators. It focuses on creating tools that handle routine tasks while freeing humans for higher-value creative work. It maintains a "human in the loop" philosophy, includes collaboration features, emphasizes quality improvement over cost reduction, and designs tools that supplement human expertise.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Positions tool as enhancing rather than replacing human expertise | ||||
| Maintains "human in the loop" philosophy | ||||
| Handles routine tasks while freeing users for higher-value work | ||||
| Includes collaboration features | ||||
| Emphasizes quality improvement in messaging | ||||
| Extends rather than replaces human expertise | ||||
| Section Score |
3. Rapid Tool Development Framework
Summary: This pillar establishes standardized templates and workflows for quick tool creation, creates modular components that can be reused across different tools, designs natural language interfaces for tool specification, provides clear documentation and examples, implements CI/CD pipelines for streamlined deployment, and creates a repository of reusable UI components and design patterns.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Uses standardized templates | ||||
| Employs modular, reusable components | ||||
| Follows project coding standards | ||||
| Includes comprehensive documentation | ||||
| Deployed through standard CI/CD pipeline | ||||
| Uses shared UI components and patterns | ||||
| Section Score |
4. Educational Pedagogy Integration
Summary: This pillar ensures tools align with established learning science principles and promote active learning, engagement, and accessibility. It supports diverse teaching and learning styles, facilitates assessment and feedback mechanisms, incorporates Universal Design for Learning (UDL) principles, enables mapping of content to learning objectives, and supports various educational contexts (higher ed, K-12, corporate training).
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Aligns with established learning science principles | ||||
| Promotes active learning and engagement | ||||
| Supports diverse teaching and learning styles | ||||
| Facilitates assessment and feedback | ||||
| Incorporates Universal Design for Learning principles | ||||
| Enables mapping to learning objectives | ||||
| Supports various educational contexts | ||||
| Section Score |
5. Faculty Empowerment & Agency
Summary: This pillar designs intuitive interfaces requiring minimal technical expertise while providing granular control over AI-generated outputs. It enables customization to match specific teaching contexts and disciplines, creates clear documentation, respects faculty expertise and pedagogical autonomy, provides multiple pathways for faculty with different comfort levels with technology, and allows faculty to contribute to tool improvement.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Provides intuitive interface requiring minimal technical expertise | ||||
| Offers granular control over AI-generated outputs | ||||
| Allows customization for specific teaching contexts | ||||
| Includes clear, jargon-free documentation | ||||
| Respects faculty expertise and autonomy | ||||
| Provides multiple pathways for different tech comfort levels | ||||
| Enables user contributions to tool improvement | ||||
| Section Score |
6. Ethical AI Implementation
Summary: This pillar ensures transparency about AI involvement in content creation, addresses bias in AI-generated content and recommendations, protects user privacy and data security, provides options for faculty to review and modify AI suggestions, establishes clear guidelines for appropriate use cases, implements content filtering for problematic outputs, and creates governance structures for ethical oversight.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Transparent about AI involvement in content creation | ||||
| Addresses bias in AI-generated content | ||||
| Protects user privacy and data security | ||||
| Provides options to review and modify AI suggestions | ||||
| Establishes clear guidelines for appropriate use | ||||
| Implements content filtering for problematic outputs | ||||
| Section Score |
7. Institutional Integration
Summary: This pillar designs tools to work within existing educational technology ecosystems, creates flexible authentication and authorization systems, supports institution-specific branding and customization, facilitates analytics that align with institutional assessment needs, complies with institutional data policies and regulations, enables integration with Learning Management Systems, and provides deployment options that work with various IT policies.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Works within existing educational technology ecosystems | ||||
| Supports flexible authentication | ||||
| Allows institution-specific branding/customization | ||||
| Facilitates analytics aligned with institutional needs | ||||
| Complies with institutional data policies | ||||
| Enables LMS integration where appropriate | ||||
| Provides flexible deployment options | ||||
| Section Score |
8. Continuous Improvement Through User Feedback
Summary: This pillar implements robust feedback mechanisms in every tool, creates clear pathways for feature requests and bug reporting, establishes analytics to identify usage patterns and pain points, maintains regular communication with the user community, conducts usability testing throughout development, prioritizes improvements based on user impact, and closes the feedback loop by communicating changes made in response to feedback.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Includes robust feedback mechanisms | ||||
| Provides clear pathways for feature requests | ||||
| Incorporates analytics to identify usage patterns | ||||
| Maintains communication channels with users | ||||
| Has undergone usability testing | ||||
| Demonstrates improvements based on user feedback | ||||
| Section Score |
9. Scalable and Sustainable Infrastructure
Summary: This pillar designs for growth in both users and functionality, considers resource usage and optimization, creates clear maintenance documentation and procedures, establishes support systems that can scale with adoption, implements monitoring and alerting for system health, designs with cost efficiency in mind, and plans for long-term sustainability beyond initial development.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Designed for growth in users and functionality | ||||
| Optimized for resource efficiency | ||||
| Includes maintenance documentation | ||||
| Has appropriate monitoring and alerting | ||||
| Designed with cost efficiency in mind | ||||
| Plans for long-term sustainability | ||||
| Section Score |
10. Knowledge Sharing and Community Building
Summary: This pillar facilitates sharing of best practices among users, creates opportunities for collaborative tool development, documents case studies and success stories, builds mechanisms for peer support and mentoring, hosts regular community events and webinars, recognizes and celebrates community contributions, and fosters a diverse and inclusive community of practice.
| Criteria | DefMatch | App 2 | App 3 | Notes |
|---|---|---|---|---|
| Facilitates sharing of best practices | ||||
| Supports collaborative development | ||||
| Includes case studies or examples | ||||
| Provides mechanisms for peer support | ||||
| Recognizes community contributions | ||||
| Promotes inclusive community of practice | ||||
| Section Score |
Overall Scores
| Pillar | DefMatch | App 2 | App 3 |
|---|---|---|---|
| 1. Accessibility by Design | |||
| 2. Augmentation, Not Replacement | |||
| 3. Rapid Tool Development Framework | |||
| 4. Educational Pedagogy Integration | |||
| 5. Faculty Empowerment & Agency | |||
| 6. Ethical AI Implementation | |||
| 7. Institutional Integration | |||
| 8. Continuous Improvement | |||
| 9. Scalable and Sustainable Infrastructure | |||
| 10. Knowledge Sharing and Community Building | |||
| TOTAL SCORE | |||
| PERCENTAGE |
Scoring Guide
- 90-100%: Exemplary - Fully aligned with VIDA pillars
- 75-89%: Strong - Well-aligned with most VIDA pillars
- 60-74%: Satisfactory - Meets essential VIDA requirements
- 40-59%: Needs Improvement - Addresses some VIDA pillars but has significant gaps
- Below 40%: Redesign Recommended - Substantial misalignment with VIDA pillars
Action Items
| Priority | Issue | Recommendation | Assigned To | Due Date |
|---|---|---|---|---|