VIDA Project Agents - RutgersGRID/VIDAHub GitHub Wiki

VIDA Project Agents

This page provides an overview of the various AI-powered agents being developed as part of the VIDA (Virtual Instructional Design Assistant) project. Each agent addresses specific instructional design needs and is developed within its own repository.

Agent Categories and Overview

VIDA agents are organized into the following categories to reflect their primary purpose and functionality:

Category Description Agents
Content Creation Tools that help generate educational content Learning Objectives Agent, Syllabus Creator, Document Generator
Accessibility Tools focused on making content accessible to all learners ALLY Assistant, Alt Tag Generator, Text-to-Speech, Speech-to-Text
Interactive Learning Tools that create engaging, interactive learning experiences Definition Match Game, Educational Game Creator, GRID Visual Novel Engine
Analysis & Intelligence Tools that analyze content and provide insights Multi-Modal Educational Assistant, Document Analysis, Video/Image Analysis
Communication Tools that facilitate student-teacher communication HUHY - Help Us Help You, Chatbot Pattern, Voice Agent
Information Retrieval Tools that help find and organize information RAG Pattern Agent

Currently, VIDA has 6 agents in active development and 11 planned agents across these categories, with a focus on Phase 1 implementation of foundational tools.

Current Agents

HUHY - Help Us Help You (TOP PRIORITY)

  • Category: Communication
  • Repository: [Coming Soon]
  • Description: A sophisticated assistant that guides faculty members through identifying and describing their specific needs, allowing UOES (University Office of Educational Services) to more effectively support them. This tool serves as a needs assessment and service request interface that helps faculty articulate their challenges and requirements.
  • Key Features:
    • Interactive needs assessment questionnaire
    • AI-guided problem identification workflow
    • Service categorization and matching
    • Request prioritization system
    • Integration with UOES service ticketing
    • Faculty request tracking dashboard
    • Follow-up and feedback collection
  • Status: Planned - Phase 1 (TOP PRIORITY)
  • Integration Points: UOES ticketing system, other VIDA tools
  • Technical Stack: Python, Streamlit, NLP for request analysis
  • Lead Developer: TBD

ALLY Assistant (TOP PRIORITY)

  • Category: Accessibility
  • **Agent Wiki Page: Ally Assistant on wiki
  • Repository: [Coming Soon]
  • Description: An agent you can copy the ALLY tool report into and have it guide you through making the suggested changes. (was: An agent that integrates with Canvas' ALLY tool to help educators improve the accessibility of their course content. This assistant provides automated accessibility analysis, remediation recommendations, and alternative format creation for educational materials.)
  • Key Features:
    • Integration with Canvas ALLY accessibility checker
    • Automated accessibility analysis of documents, PDFs, images, and other content
    • Remediation recommendations with step-by-step guidance
    • Alternative format generation (HTML, audio, electronic braille, etc.)
    • Accessibility score tracking and improvement metrics
    • Batch processing capabilities for multiple files
  • Status: Planned - Phase 1 (TOP PRIORITY)
  • Integration Points: Canvas LMS, other VIDA accessibility tools
  • Technical Stack: Python, Streamlit, Canvas API, ALLY API
  • Lead Developer: TBD

Voice Agent

  • Category: Communication
  • Repository: Voice Agent Repository
  • Description: The Voice Agent provides speech-to-text and text-to-speech capabilities for educators, enabling more accessible content creation and consumption. It can be integrated into various VIDA tools to add voice interaction capabilities.
  • Key Features:
    • Audio transcription for lecture recordings
    • Text-to-speech for creating accessible audio versions of text content
    • Voice command interface for hands-free operation of VIDA tools
    • Pronunciation guidance for language learning applications
  • Status: Under development - Phase 1
  • Integration Points: Can be integrated with Syllabus Creator, Alt Tag Generator, and other VIDA tools
  • Technical Stack: Python, Hugging Face speech models, Streamlit for interface
  • Lead Developer: Rick Anderson

Multi-Modal Educational Assistant

  • Category: Analysis & Intelligence
  • Repository: Multi-Modal Education Assistant Repo link
  • Streamlit Cloud: Link to app on streamlit
  • Forked GitHub Repo: Coming Soon
  • Evaluation Link: Coming Soon
  • Description: A comprehensive agent that combines multiple AI capabilities to assist educators with content creation, analysis, and transformation across different modalities.
  • Key Features:
    • Document Analysis and extraction
    • Document Generation
    • Text-to-Speech conversion
    • Speech-to-Text transcription
    • Alt tag generation for images
    • Video and Image Analysis
  • Model Implementation:
    • Hugging Face - SmolLVM2 (Working)
    • Anthropic – (In progress)
    • Hugging Face – Zephyr-7b-beta (In Progress)
    • Hugging Face – Microsoft/Phi-2 (Working)
  • Technical Highlights:
    • Multi-model architecture allows selecting the optimal model for specific tasks
    • Unified chat interface with context retention between models
    • Builds upon previous chatbot project architecture
    • Model switching for task optimization
  • Status: Under development - Phase 1
  • Lead Developer: Bryan Zunigas

Definition Match Game Agent

Learning Objectives Agent (Top Priority)

  • Category: Content Creation
  • Repository: Link to repo, V2
  • Streamlit Cloud - Link
  • Agent Evaluation - Link
  • Description: An agent that helps educators create, align, and manage learning objectives for their courses, ensuring proper alignment with curriculum standards and providing suggestions for assessment methods.
  • Status: Under development - Phase 1
  • Lead Developer: Maka Gradin

Planned Agents

Alt Tag Generator Agent (Top Priority)

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: An agent focused on generating appropriate alternative text descriptions for images to improve content accessibility.
  • Status: Planned - Phase 1

Syllabus Creator Agent

  • Category: Content Creation
  • Repository: [Coming Soon]
  • Description: An agent designed to help faculty create comprehensive, well-structured course syllabi with minimal effort.
  • Status: Planned - Phase 1

Educational Game Creator Agent

  • Category: Interactive Learning
  • Repository: [Coming Soon]
  • Description: An agent that enables educators to design engaging educational games aligned with learning objectives.
  • Status: Planned - Phase 2

Character Creator Agent

  • Category: Interactive Learning
  • Repository: [Coming Soon]
  • Description: An agent for generating engaging characters for educational narratives and scenarios.
  • Status: Planned - Phase 2

Chatbot Pattern Agent

  • Category: Communication
  • Repository: [Coming Soon]
  • Description: A reusable pattern for creating educational chatbots that can be customized for different instructional contexts, providing conversational support for students.
  • Status: Planned - Phase 2

RAG Pattern Agent

  • Category: Information Retrieval
  • Repository: [Coming Soon]
  • Description: An agent implementing Retrieval-Augmented Generation (RAG) for context-aware information retrieval and response generation based on educational content repositories.
  • Status: Planned - Phase 2

Text-to-Speech Agent

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: An agent that converts written text to natural-sounding speech, enhancing accessibility and creating audio versions of educational content.
  • Status: Planned - Phase 2

Speech-to-Text Agent

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: An agent that accurately transcribes spoken language to text, enabling voice-based input and content creation for educators.
  • Status: Planned - Phase 2

Vision to Alt Tags Agent (Top Priority)

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: An agent that uses computer vision models to analyze images and automatically generate appropriate alternative text descriptions, improving accessibility and saving time for educators.
  • Status: Planned - Phase 2

Video Summarization Agent (Top Priority)

  • Category: Analysis & Intelligence
  • Repository: [Coming Soon]
  • Description: An agent that analyzes educational videos to create concise summaries, chapter markers, and key point extraction, making video content more accessible and easier to navigate.
  • Status: Planned - Phase 2

GRID Visual Novel Engine Agent

  • Category: Interactive Learning
  • Repository: [Coming Soon]
  • Description: An agent that assists educators in creating educational visual novels and interactive narratives using a spreadsheet-based approach, generating game dialog, story branches, character interactions, and exporting to a playable format.
  • Status: Planned - Phase 2

Agent Development Roadmap

Phase 1 (Months 1-4)

  • HUHY - Help Us Help You - Initial Development (TOP PRIORITY)
  • ALLY Assistant - Initial Development (TOP PRIORITY)
  • Voice Agent - Initial Development
  • Alt Tag Generator Agent - Initial Development
  • Syllabus Creator Agent - Planning
  • Definition Match Game Agent - Advanced Features
  • Learning Objectives Agent - Advanced Features

Phase 2 (Months 5-8)

  • HUHY - Help Us Help You - Enhanced Features
  • ALLY Assistant - Enhanced Features
  • Voice Agent - Enhanced Features
  • Alt Tag Generator Agent - Enhanced Features
  • Syllabus Creator Agent - Initial Development
  • Educational Game Creator Agent - Initial Development
  • Character Creator Agent - Planning
  • Chatbot Pattern Agent - Initial Development
  • RAG Pattern Agent - Planning
  • Text-to-Speech Agent - Initial Development
  • Speech-to-Text Agent - Initial Development
  • Vision to Alt Tags Agent - Planning
  • Video Summarization Agent - Planning
  • Definition Match Game Agent - Enhanced Features
  • Learning Objectives Agent - Enhanced Features
  • GRID Visual Novel Engine Agent - Planning

Phase 3 (Months 9-12+)

  • All agents - Advanced features and integration
  • RAG Pattern Agent - Initial Development
  • Vision to Alt Tags Agent - Initial Development
  • Video Summarization Agent - Initial Development
  • Definition Match Game Agent - Initial Development
  • Learning Objectives Agent - Initial Development
  • GRID Visual Novel Engine Agent - Initial Development
  • User-created agent specification platform

Contributing to Agent Development

If you're interested in contributing to any of these agents, please follow these steps:

  1. Check the specific agent's repository for open issues
  2. Review the Contributing Guidelines
  3. Set up your development environment following the Getting Started Guide
  4. Pick an issue to work on or propose a new feature
  5. Submit a pull request following our workflow process

Agent Integration Framework

All VIDA agents follow a common integration framework to ensure they can work together seamlessly. This includes:

  • Standardized API endpoints
  • Common authentication mechanisms
  • Consistent data exchange formats
  • Shared UI components and patterns
  • Unified deployment infrastructure

For more details on the technical implementation of our agent integration framework, see the Technical Architecture documentation.

Last updated: May 20, 2025