1MEIC01T4: AccessAI - FEUP-MEIC-DS-2024-25/ai4sd GitHub Wiki
The Access AI project essentially developed an artificial intelligence-based tool for the assessment and improvement of the accessibility, inclusiveness, and usability of software applications. Access AI provides developers with realistic feedback to ensure applications are designed to accommodate users with a variety of disabilities, such as visual, motor, and cognitive disabilities.
Access AI is integrated into development processes to help developers create more user-centric and inclusive software, thanks to the latest artificial intelligence technologies and its adherence to established accessibility standards. It serves as a critical resource for integrating accessibility considerations throughout the software development lifecycle.
Vision
The product vision of Access AI is to nurture a digital environment that is inclusive, where individuals of all abilities can interact with web applications without encountering friction. We believe technology should be used to lift barriers, not create them. Access AI uses the power of state-of-the-art artificial intelligence to automatically find accessibility problems, offer actionable suggestions on how to resolve those problems, and then guide best practices for inclusion.
The assistant is intuitive and user-friendly, to help developers without specialized training improve accessibility features. It guarantees equal access to information and services for all users, providing clear and actionable recommendations.
In a nutshell, Access AI aims at accessibility being one of the fundamental parts of user experience design in order to make everyone feel valued and included in the digital space.
Research
Applitools is a tool that enables accessibility validation, focusing on the standards that define minimum contrast ratios for text and graphics. It allows developers to ensure that visual elements meet the minimum requirements of accessibility. While Applitools is effective for tool for visual validation, it does not support detection of missing alternative text, generation of colorblind-friendly colors or detection of options to adjust font size, among other accessibility validations, which are essential for a complete accesibility evaluation.
Another tool we found is axe Accessibility Linter, which checks code files for common accessibility defects, supporting various frameworks such as React (JSX), React Native, Angular, Vue, HTML, and Markdown. The axe Linter helps developers identify accessibility issues during coding, providing immediate feedback on potential barriers for users with disabilities. Despite this fact, it still lacks interactivity, since it does not provide a conversational interface or real-time support through natural language prompts, limiting its usability for developers to seek communication about the guidelines on improving their accessibility.
In contrast, our AI assistant for accessibility evaluation offers real-time accessibility feedback alongside a conversational interface, enabling developers to ask natural language questions about accessibility concerns. Unlike Applitools and axe Accessibility Linter, this assistant will provide total support all over the different accessibility barriers: visual, auditory, and mobility-related impairments, along with tools like the colorblind-friendly palette generator. By offering interactive guidance and personalized recommendations, our assistant will support developers in creating inclusive applications with fewer limitations than current tools.
Domain Analysis
Architecture and design
Technologies
When planning the development of the AI-powered accessibility assistant, we identified several core technologies and frameworks that fit the project’s needs. These decisions were made by the development team based on project requirements. Below is an overview of the main technologies and their roles:
Python
- The backend of the project is built using Python with the Flask web framework. Flask provides a lightweight and flexible environment for:
- Hosting the AI processing logic
- Providing API endpoints to handle user requests
- Integrating with the Google Generative AI library to process accessibility prompts
- Ensuring cross-origin communication with the frontend using Flask-CORS
Google Gemini AI
- The project incorporates Google Gemini AI as its core artificial intelligence platform to evaluate and generate accessibility reports based on predefined prompts. These prompts are mainly focused on:
- Vision Accessibility: Addressing needs for users with visual impairments
- Motor Accessibility: Ensuring usability for individuals with motor limitations
- Usability Accessibility: Checking for error prevention and user-friendly design
Node.js and TypeScript
- The project’s frontend, built as a VSCode extension, utilizes:
- Node.js: Provides the runtime environment for developing and running the extension
- TypeScript: Used for writing strongly typed code to build a robust extension functionality
- NPM: Manages dependencies for building and testing the extension
VSCode API
- The project integrates with Visual Studio Code through its API. The extension uses:
- Webview API: For creating interactive user interfaces within the VSCode editor
- Command Registration: To define custom commands, like opening the chatbot panel
- Event Handling: To respond to user actions and extension events dynamically
Docker
- The project is containerized using Docker, ensuring a consistent development environment.
Git
- Git is used for version control, ensuring smooth collaboration
Prototype Overview
In Sprint 0, we developed two prototypes that show the main features we plan to implement. Both prototypes include a chat interface and a colorblind-friendly palette generator.
The first prototype provides a complete accessibility evaluation, assessing the software's usability across multiple impairments, including colorblindness, blindness, deafness, and reduced mobility.
The second prototype shows an interactive chat box, where developers can ask accessibility-related questions, such as whether their software includes captions for audio content.
Additionally, both prototypes feature a colorblind-friendly palette generator, designed to assist developers in creating accessible color schemes. A developer can specify the number of colors, the type of colorblindness to customize the palette for, and input brand colors. The tool then generates a customized palette that ensures better accessibility for users with color vision deficiencies.
Development guide
Our assistant can evaluate accessibility in HTML, CSS, and JavaScript files. To perform these evaluations, the files the user wants to evaluate must be placed in the testfile folder. This folder is located within the gemini directory, which is inside the src folder, nested under the project's root directory, access-ai. The full path is: /access-ai/src/gemini/testfile. All files within this folder will be evaluated, provided they are in HTML, CSS, or JavaScript formats, as specified earlier.
To use our assistant, the user needs to open VSCode in the src folder located inside the access-ai directory /access-ai/src. Next, run the Docker container by executing the command docker compose up in the terminal. Once Docker is running, press F5 to launch a new VSCode window. In this new window, the user needs to press CTRL + SHIFT + P to open the command palette, then search for "Access-ai" in the search bar to open the chatbox and start interacting with the AI assistant.
Security concerns
Identify potential security vulnerabilities classes and explain what the team has done to mitigate them.
Quality assurance
Each pull request goes under a review by two team members, excluding the original developer, providing an additional layer of quality control in the development workflow. While formal tests were not created, the application was rigorously tested throughout the development phase so that we could be sure when checking some feature as done. All developers were responsible for testing the application, and the Product Owner (PO) verified that the results met expectations.
How to use
After completing the section Development Guide, the user should have the extension running with the AI assistant fully operational. The following video demonstrates how to use the AccessAI assistant:
https://github.com/user-attachments/assets/90a3dcf8-1ecb-4fb4-b725-a6ca4be0d3fd
How to contribute
Explain what a new developer should know in order to develop the tool, including how to build, run and test it in a development environment.
Defer technical details to the technical documentation below, which should include information and decisions on architectural, design and technical aspects of the tool.
Sprint 1 Retrospective
What went well?
The team has kept the habit, started before sprint 1, of having weekly meetings to discuss progress, assign tasks and encourage the discussion of ideas with the group. These meetings always have a fun and informal tone and environment which promotes team building. All of this had supported a productive workflow and a positive team dynamic. During this sprint, we were able to integrate VSCode Gemini and develop a script for prompt testing.
What went poorly?
Even though we were able to integrate VSCode Gemini and develop a script for prompt testing, this was still not integrated on the main branch.
What can we do to improve?
We also started developing some of the features for our accessibility assistant, which are still in review because they were not tested. So, in addition to the integration on the main branch, we intend to continue developing and test features for our acessibility assistant.
What ideas do we have?
On the next sprint, we expect to maintain the positive practises that helped the team until now, and complete the tasks left from this sprint (integration on main and feature testing). We also count on advancing in more features development, but only after ensuring everything already started is done and acessing the time we may have left.
Sprint 2 Retrospective
What went well?
The team has kept the habit, started before sprint 1, of having weekly meetings to discuss progress, assign tasks and encourage the discussion of ideas with the group. These meetings always have a fun and informal tone and environment which promotes team building. All of this had supported a productive workflow and a positive team dynamic. During this sprint, we were able to start integrating the extension on the main branch and some features testing.
What went poorly?
Due to the heavy work load, we did not implement any new features.
What can we do to improve?
We can try to improve the organization of each team member's tasks by aligning them with their available time and skills. This approach ensures efficient allocation of resources in order to implement new features.
What ideas do we have?
On the next sprint, we expect to maintain the positive practises that helped the team until now, and complete the tasks left from this sprint. We will be completing and refining what we already have focusing, prioritizing the quality of the product instead of the quantity of fetaures.
Sprint 3 Retrospective
What went well?
During this sprint, we were able to complete and refine what we had planned for this spint. We integrated the extension on the main branch and implemented and tested the features we thought would be most useful to potential users.
What went poorly?
Due to time constraints, the weekly meetings that were done in the last to sprints did not happen, which at first create some communication problems. However, the team was able to overcome those problems in order to assure the progress of the project.
Final Scrum Board