VIDA Project Agents - RutgersGRID/VIDAHub GitHub Wiki

VIDA Project Agents

This page provides an overview of the various AI-powered agents being developed as part of the VIDA (Virtual Instructional Design Assistant) project. Each agent addresses specific instructional design needs and is developed within its own repository.

Top Priority Agents (Phase 1)

ALLY Assistant (TOP PRIORITY)

  • Category: Accessibility
  • Agent Wiki Page: Ally Assistant on wiki
  • Repository: [Coming Soon]
  • Description: An agent you can copy the ALLY tool report into and have it guide you through making the suggested changes. This assistant provides automated accessibility analysis, remediation recommendations, and alternative format creation for educational materials.
  • Key Features:
    • Integration with Canvas ALLY accessibility checker
    • Automated accessibility analysis of documents, PDFs, images, and other content
    • Remediation recommendations with step-by-step guidance
    • Alternative format generation (HTML, audio, electronic braille, etc.)
    • Accessibility score tracking and improvement metrics
    • Batch processing capabilities for multiple files
  • Supporting Development Agents:
    • Alt Tag Generator Agent - Will provide image accessibility capabilities for ALLY
    • Video Summarization Agent - Will contribute video accessibility features for ALLY
    • Vision to Alt Tags Agent - Will provide advanced computer vision-based accessibility analysis for ALLY
    • Canvas Backup Course Explorer - Provides foundation for Canvas integration and ALLY report processing
  • Status: Planned - Phase 1 (TOP PRIORITY)
  • Integration Points: Canvas LMS, other VIDA accessibility tools
  • Technical Stack: Python, Streamlit, Canvas API, ALLY API
  • Lead Developer: TBD

Learning Objectives Agent (TOP PRIORITY)

  • Category: Content Creation
  • Monday Project: Link
  • Repository: Link to repo, V2
  • Streamlit Cloud: Link
  • Agent Evaluation: Link
  • Description: An agent that helps educators create, align, and manage learning objectives for their courses, ensuring proper alignment with curriculum standards and providing suggestions for assessment methods.
  • Status: Under development - Phase 1
  • Lead Developer: Maka Gradin

Alt Tag Generator Agent (TOP PRIORITY - ALLY Supporting Agent)

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: An agent focused on generating appropriate alternative text descriptions for images to improve content accessibility. This agent's development will directly support the ALLY Assistant by providing core image accessibility capabilities.
  • ALLY Integration: Core functionality will be integrated into ALLY Assistant for automated alt tag generation
  • Status: Planned - Phase 1

Vision to Alt Tags Agent (TOP PRIORITY - ALLY Supporting Agent)

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: An agent that uses computer vision models to analyze images and automatically generate appropriate alternative text descriptions, improving accessibility and saving time for educators. This agent's development will directly support the ALLY Assistant by providing advanced computer vision-based accessibility capabilities.
  • ALLY Integration: Advanced vision processing capabilities will be integrated into ALLY Assistant for comprehensive image accessibility analysis
  • Status: Planned - Phase 2

Video Summarization Agent (TOP PRIORITY - ALLY Supporting Agent)

  • Category: Analysis & Intelligence
  • Repository: [Coming Soon]
  • Description: An agent that analyzes educational videos to create concise summaries, chapter markers, and key point extraction, making video content more accessible and easier to navigate. This agent's development will directly support the ALLY Assistant by providing video accessibility and content analysis capabilities.
  • ALLY Integration: Video accessibility features will be integrated into ALLY Assistant for comprehensive multimedia accessibility support
  • Status: Planned - Phase 2

Agent Categories and Overview

VIDA agents are organized into the following categories to reflect their primary purpose and functionality:

Category Description Agents
Content Creation Tools that help generate educational content Learning Objectives Agent, Syllabus Creator, Document Generator
Accessibility Tools focused on making content accessible to all learners ALLY Assistant, Alt Tag Generator, Speech Agent
Interactive Learning Tools that create engaging, interactive learning experiences Educational Game Creator, GRID Visual Novel Engine
Analysis & Intelligence Tools that analyze content and provide insights Video Summarization Agent, Canvas Backup Course Explorer, Document Analysis
Communication Tools that facilitate student-teacher communication Chatbot Pattern
Information Retrieval Tools that help find and organize information RAG Pattern Agent
Experimental Proof-of-concept agents exploring various approaches Multi-Modal Educational Assistant, Voice Agent, Definition Match Game

Currently, VIDA has 1 agent in active development (Canvas Backup Course Explorer), 3 experimental agents, and 8 planned agents across these categories, with a focus on Phase 1 implementation of foundational tools and ALLY Assistant supporting development.

Current Agents in Development

Canvas Backup Course Explorer

  • Category: Analysis & Intelligence / Canvas Integration
  • Wiki Details: Canvas Explorer
  • Repository: GitHub Repo
  • Created: June 21, 2025
  • Description: A foundational agent that provides Canvas course backup functionality with integrated editing, chat capabilities, and ALLY report ingestion. This agent serves as a technical foundation for ALLY Assistant development and Canvas LMS integration.
  • Key Features:
    • Canvas course backup and data extraction
    • Course content editing capabilities
    • Chat interface for course interaction
    • ALLY accessibility report ingestion and processing
    • Foundation for Canvas API integration patterns
  • ALLY Integration: Provides core Canvas integration framework and ALLY report processing capabilities that will be incorporated into ALLY Assistant
  • Status: Under development - Phase 1 (Supporting ALLY development)
  • Lead Developer: Rick Anderson

Experimental Agents

These agents were developed as proof-of-concept and experimental implementations to explore various educational technology approaches:

Multi-Modal Educational Assistant (EXPERIMENTAL)

  • Category: Analysis & Intelligence
  • Repository: Multi-Modal Education Assistant Repo link
  • Streamlit Cloud: Link to app on streamlit
  • Forked GitHub Repo: Coming Soon
  • Evaluation Link: Coming Soon
  • Description: A comprehensive experimental agent that combines multiple AI capabilities to assist educators with content creation, analysis, and transformation across different modalities.
  • Key Features:
    • Document Analysis and extraction
    • Document Generation
    • Text-to-Speech conversion
    • Speech-to-Text transcription
    • Alt tag generation for images
    • Video and Image Analysis
  • Model Implementation:
    • Hugging Face - SmolLVM2 (Working)
    • Anthropic – (In progress)
    • Hugging Face – Zephyr-7b-beta (In Progress)
    • Hugging Face – Microsoft/Phi-2 (Working)
  • Technical Highlights:
    • Multi-model architecture allows selecting the optimal model for specific tasks
    • Unified chat interface with context retention between models
    • Builds upon previous chatbot project architecture
    • Model switching for task optimization
  • Status: Experimental - Phase 1
  • Lead Developer: Bryan Zunigas

Voice Agent (EXPERIMENTAL)

  • Category: Communication
  • Repository: Voice Agent Repository
  • Description: An experimental agent that provides speech-to-text and text-to-speech capabilities for educators, enabling more accessible content creation and consumption. It can be integrated into various VIDA tools to add voice interaction capabilities.
  • Key Features:
    • Audio transcription for lecture recordings
    • Text-to-speech for creating accessible audio versions of text content
    • Voice command interface for hands-free operation of VIDA tools
    • Pronunciation guidance for language learning applications
  • Status: Experimental - Phase 1
  • Integration Points: Can be integrated with Syllabus Creator, Alt Tag Generator, and other VIDA tools
  • Technical Stack: Python, Hugging Face speech models, Streamlit for interface
  • Lead Developer: Rick Anderson

Definition Match Game Agent (EXPERIMENTAL)

Planned Agents

Syllabus Creator Agent

  • Category: Content Creation
  • Repository: [Coming Soon]
  • Description: An agent designed to help faculty create comprehensive, well-structured course syllabi with minimal effort.
  • Status: Planned - Phase 1

Educational Game Creator Agent

  • Category: Interactive Learning
  • Repository: [Coming Soon]
  • Description: An agent that enables educators to design engaging educational games aligned with learning objectives.
  • Status: Planned - Phase 2

Character Creator Agent

  • Category: Interactive Learning
  • Repository: [Coming Soon]
  • Description: An agent for generating engaging characters for educational narratives and scenarios.
  • Status: Planned - Phase 2

Chatbot Pattern Agent

  • Category: Communication
  • Repository: [Coming Soon]
  • Description: A reusable pattern for creating educational chatbots that can be customized for different instructional contexts, providing conversational support for students.
  • Status: Planned - Phase 2

RAG Pattern Agent

  • Category: Information Retrieval
  • Repository: [Coming Soon]
  • Description: An agent implementing Retrieval-Augmented Generation (RAG) for context-aware information retrieval and response generation based on educational content repositories.
  • Status: Planned - Phase 2

Speech Agent (Text-to-Speech & Speech-to-Text)

  • Category: Accessibility
  • Repository: [Coming Soon]
  • Description: A unified agent that provides comprehensive speech processing capabilities, converting written text to natural-sounding speech and accurately transcribing spoken language to text, enhancing accessibility and enabling voice-based input for educators.
  • Key Features:
    • Text-to-speech conversion for creating accessible audio versions of educational content
    • Speech-to-text transcription for voice-based input and content creation
    • Multiple voice options and language support
    • Audio quality optimization for educational contexts
    • Integration with other VIDA accessibility tools
  • Status: Planned - Phase 2

GRID Visual Novel Engine Agent

  • Category: Interactive Learning
  • Repository: [Coming Soon]
  • Description: An agent that assists educators in creating educational visual novels and interactive narratives using a spreadsheet-based approach, generating game dialog, story branches, character interactions, and exporting to a playable format.
  • Status: Planned - Phase 2

Deprioritized Agents

HUHY - Help Us Help You (DEPRIORITIZED)

  • Category: Communication
  • Repository: [Coming Soon]
  • Description: A sophisticated assistant that guides faculty members through identifying and describing their specific needs, allowing UOES (University Office of Educational Services) to more effectively support them. This tool serves as a needs assessment and service request interface that helps faculty articulate their challenges and requirements.
  • Key Features:
    • Interactive needs assessment questionnaire
    • AI-guided problem identification workflow
    • Service categorization and matching
    • Request prioritization system
    • Integration with UOES service ticketing
    • Faculty request tracking dashboard
    • Follow-up and feedback collection
  • Deprioritization Rationale: After review, this agent was deprioritized because it is not quite relevant to the way instructional designers work, may not be relevant to our goals of consultation and creating tools, and might be overkill for academic media. Current approaches such as get-to-know-you meetings or consultation booking make more sense as a first point of contact rather than an automated system.
  • Status: Deprioritized
  • Integration Points: UOES ticketing system, other VIDA tools
  • Technical Stack: Python, Streamlit, NLP for request analysis
  • Lead Developer: TBD

Agent Development Roadmap

Phase 1 (Months 1-4) - Current Focus

  • ALLY Assistant - Initial Development (TOP PRIORITY)
  • Canvas Backup Course Explorer - Foundation development for ALLY (Supporting Agent)
  • Learning Objectives Agent - Advanced Features (TOP PRIORITY)
  • Alt Tag Generator Agent - Initial Development (TOP PRIORITY - ALLY Supporting Agent)
  • Syllabus Creator Agent - Planning
  • Experimental Agents - Multi-Modal Educational Assistant, Voice Agent, Definition Match Game (proof-of-concept)

Phase 2 (Months 5-8)

  • ALLY Assistant - Enhanced Features (incorporating Alt Tag Generator and Video Summarization capabilities)
  • Learning Objectives Agent - Enhanced Features
  • Video Summarization Agent - Initial Development (TOP PRIORITY - ALLY Supporting Agent)
  • Vision to Alt Tags Agent - Initial Development (TOP PRIORITY)
  • Alt Tag Generator Agent - Enhanced Features (integration with ALLY)
  • Canvas Backup Course Explorer - Enhanced Features (full Canvas integration)
  • Syllabus Creator Agent - Initial Development
  • Educational Game Creator Agent - Initial Development
  • Character Creator Agent - Planning
  • Chatbot Pattern Agent - Initial Development
  • RAG Pattern Agent - Planning
  • Speech Agent - Initial Development (Text-to-Speech & Speech-to-Text combined)
  • GRID Visual Novel Engine Agent - Planning

Phase 3 (Months 9-12+)

  • All agents - Advanced features and integration
  • RAG Pattern Agent - Initial Development
  • GRID Visual Novel Engine Agent - Initial Development
  • User-created agent specification platform
  • Experimental Agent Evolution - Determine which experimental agents to productionize based on feedback and usage

Contributing to Agent Development

If you're interested in contributing to any of these agents, please follow these steps:

  1. Check the specific agent's repository for open issues
  2. Review the Contributing Guidelines
  3. Set up your development environment following the Getting Started Guide
  4. Pick an issue to work on or propose a new feature
  5. Submit a pull request following our workflow process

Agent Integration Framework

All VIDA agents follow a common integration framework to ensure they can work together seamlessly. This includes:

  • Standardized API endpoints
  • Common authentication mechanisms
  • Consistent data exchange formats
  • Shared UI components and patterns
  • Unified deployment infrastructure

For more details on the technical implementation of our agent integration framework, see the Technical Architecture documentation.

Last updated: June 23, 2025