AI-Plugins for OS-DPI

Welcome!

With the rapid development of AI, we believe these technologies can significantly enhance the tools we’re building for AAC users.

The OS-DPI platform is designed for researchers exploring how user interfaces can support AAC users. To further empower these efforts, we’ve developed a suite of AI-powered plugin tools. These tools can either:

• Be integrated into your current UI, or
• Function as standalone features.

✨ Key Features

We currently provide four core AI features:

1. Text-based Keyword Prediction
2. Text-based Utterance Prediction
3. Image-based Keyword Prediction
4. Image-based Utterance Prediction

Each plugin is designed to enhance accessibility and communication for AAC users. Below, you'll find detailed explanations and examples for each feature.

1. 📚 Text-Based Keyword Prediction

Use one or more input keywords to predict related words.
This feature is useful for auto-completing keywords in a context-aware manner.

Example:
Input: birthday
Predicted keywords: party, cake, celebration

This feature can also be used as input for the Text-Based Utterance Prediction tool.

2. 🗣️ Text-Based Utterance Prediction

Input a few keywords and receive full predicted utterances (e.g., comments or questions) based on them.

Example:
Input keywords: pumpkin carve
Click Predict Questions
Generated output:

• "How to carve a pumpkin?"
• Plus 5 other relevant suggestions

This allows for communication with minimal typing effort.

3. 🖼️ Image-Based Keyword Prediction

Upload an image to automatically extract keywords based on objects and context in the image.

Use case:
Upload a kitchen image and automatically detect objects like water, sink, cup, etc., without typing a word.

4. 💬 Image-Based Utterance Prediction

Upload an image to generate relevant comments or questions based on its contents.

Use case:
Share an outing photo and let the model suggest expressive phrases like:

• "Look! I am happy!"
• "What a nice outing!"

This tool helps users communicate their thoughts and emotions through visuals.

🔧 Setup & Integration Guides

We provide step-by-step documentation to help you integrate and use the plugins:

• 📥 Steps to Import Plugins
  Learn how to add plugins into your existing UI.

• 🔌 How to Connect the Backend
  Guide to setting up the backend connection with the AI model.

• 🧠 How to Customize the Prompt
  Learn how to modify the prompt used for AI generation and test your own versions.

• 🧩 How to Interact with the Plugins
  Includes plugin usage instructions, screenshots of integrations, combining plugins in a single UI, and uploading images into OS-DPI.