1MEIC02T02: SARA - FEUP-MEIC-DS-2024-25/ai4sd GitHub Wiki
Our Software Architecture Repository Analyser (SARA) analyses text-based artifacts (source code, etc.) in software repositories and automatically detects architectural patterns.
Below are more details about the SARA, including the product vision, market research and domain analysis.
Vision
In a world where software projects are being developed at an unprecedented scale, it is important to ask foundational questions when creating a new project or expanding on an existing one: What architecture best addresses this problem? or What architectural patterns am I working with?
This task can be quite challenging, especially when developers do not have complete context for the problem.
In recent years, to address this issue, software engineers have increasingly turned to AI to help interpret their projects, as it saves time and can provide a helpful, though not always precise or correct, starting point for the team. However, it can be a hassle to switch between different tools just to get these answers, especially when they may not be the most specialized AIs for addressing this question.
With these problems in mind, we’ve developed SARA, the Software Architecture Repository Analyser. Available as a VSCode extension, SARA uses Generative AI (such as GPT, Gemini, etc.) to analyse your code directly from your remote repository and deliver a report on the existing architecture.
SARA focuses on accessibility by integrating directly into your code environment, which is a valuable feature for us developers. Our product vision states:
A quick, on-the-go tool with easy access, helping you understand the architectural patterns in your code.
Research
During the planning phase of SARA, keeping in mind our idea and proposal, we researched several existing tools to better understand the current market offerings and identify gaps where our product could provide value.
-
Structure101
Pros:
- Advanced management and visualization of all code structures.Cons:
- Requires a separate interface, lacking easy access; - Focuses on a direct approach, while the flexibility of SARA can sometimes be more suitable. -
SonarQube
Pros:
- Can give good coverage on code quality and security, alongside architecture analysis; - Supports multiple languages.Cons:
- Not specifically focused on architecture; - Requires a separate server, lacking easy access. -
Lattix Architect
Pros:
- Suitable for large-scale systems, with dependency mapping and modeling.Cons:
- Requires a complex setup and different environment, lacking easy access; - Not flexible, focusing on a direct approach.
As can be seen, most common tools, although offering rich features and different approaches to analyzing code, generally lack the flexibility that Generative AI can provide, as well as ease of access, often requiring additional interfaces, servers, or setups.
Domain Analysis
This domain analysis aims to provide a clear understanding of the system's core concepts and their interactions within SARA's final product. Further details on the current project state are outlined in the next section. In this context, Nexus refers to a database that collects GitHub repositories, and Ultron refers to the LLM chosen by the user in the front end. For the cases where Ultron is not referred, the LLMs the user may choose are for Gemini and GroqCloud models.
Domain Model
This diagram outlines SARA's core components: the frontend VS Code extension, the backend API, and various backend services including a Small Language Model, a file retrieval system, and integration with external APIs.
Sequence Diagram
The diagram illustrates the step-by-step interaction between components in the SARA platform as it processes code repositories. It shows how the user initiates the process, the system authenticates with GitHub, fetches files, processes them using LLM or SLM depending on internet connectivity, and finally displays the results to the user.
Architecture and design
The component diagram below details the envisioned architecture for SARA's final product. Currently, the components Local SLM and Nexus are not yet implemented or integrated, while the remaining architecture is already operational.
This architecture was chosen to ensure that the frontend is only dependent on the SARA API, which increases reliability by isolating the frontend’s role to user interaction and fetching local data from the workspace. In the backend, the modules are organized based on their responsibilities, with their interactions coordinated through the SARA API request handler.
Technologies
For the development of SARA, we made several key technology choices to ensure a streamlined, user-friendly, and scalable solution for analyzing architectural patterns in software repositories.
Backend with Python
Python was used in the backend of SARA due to its simplicity, readability, and easy integration capabilities with external APIs. Additionally, Python's widespread use in AI and machine learning makes it easy to integrate with AI models, important for the potential integration of a Small Language Model (SLM) locally within SARA.
GitHub API Integration
We used the GitHub API to fetch repository Python files by developing SARA as a GitHub App. This approach offers fine-tuned access control over repositories, allowing us to make API requests attributed to the app rather than requiring users to share private access tokens. By authenticating as an app installation, SARA can access resources owned by the user or organization where it's installed. This makes it ideal for automation workflows, like ours, that don’t require user input.
Gemini API for Architecture Analysis
The Gemini API was used to analyse the contents of Python files and generate a detailed description of the architectural patterns detected in each code section. Although we had multiple AI models to choose from, such as OpenAI and VertexAI, we decided on Gemini due to its comprehensive and easy-to-understand documentation, which simplified the integration process. Additionally, Gemini's higher prompt token limit allowed us to upload larger code files without hitting rate limits. We also took into consideration the free plans available and the client's recommendation, making Gemini the adequate choice for the magnitude of this project.
GroqCloud API for Architectural Analysis
The GroqCloud API is an API that hosts several LLMs, which we use, like the Gemini API, to analyze the contents of Python files and generate detailed descriptions of the architectural patterns detected in each code section. We chose this API because of its variety of models, which provides users with greater flexibility in their analyses, and its free plan, which is always appealing for avoiding costs. However, it has the drawback of a lower token limit per request compared to the Gemini API.
API with Flask
For the backend, we considered several technologies, including Flask, FastAPI, and Django. Since this is a small project and only requires a functional API to support the VSCode extension, Django was ruled out due to its complexity, which would be excessive for the project’s needs. This left us with two remaining options: Flask and FastAPI. While FastAPI is designed for high-concurrency tasks and generally offers better performance, we chose Flask because it has a lower learning curve, is easier to use, and our project has low concurrency requirements.
Frontend with VSCode Extension
For the front end, we chose to develop SARA as a VSCode extension. This decision was driven by two main factors: ease of accessibility for users, who can get architectural insights directly in their workplace without switching to external applications, and easier access to code within VSCode. By embedding the app inside VSCode, we can access both the user’s GitHub repository and the code they’re actively working on.
The language chosen for the frontend was TypeScript, for multiple reasons:
- The VSCode API itself is written in TypeScript, ensuring smoother integration;
- TypeScript catches type-related errors before compilation, unlike JavaScript, while still granting access to its vast library;
- It is easier to maintain and scale when compared to other common frontend languages;
- With a large community in frontend development, TypeScript offers ample resources to assist in development.
Current State Overview
In Sprint 0 through 3, we implemented a transversal program that not only validated our technology choices but also highlighted potential challenges for the future and the level of effort required to address them, such as the complexities of interacting with LLM APIs and the effort required for handling diverse usage cases.
The project includes options for remote repository analysis and local file analysis. In the first case, SARA automatically detects the repository the user has opened in VSCode, fetches the Python files from the remote main branch, and sends them to the chosen Language Model for analysis. In the latter case, the VSCode workspace Python files are zipped and sent to SARA's backend. Currently, SARA supports analysis with Gemini or GroqCloud models. These functionalities are exposed via an API, which is called by the front end when the user requests it through the dropdown menu. After processing, a temporary markdown file opens in the workspace, containing relevant code snippets and architectural patterns identified in the analysis.
The code for our frontend and backend are available on FEUP-MEIC-DS-2024-25 AI4SD repository. Within X-Men, SARA is categorized as an Analyser.
Known Issues
Currently, the analysis of remote repositories is not functional, as we are still in the process of integrating SARA with the AI4SD Jarvis service.
Security concerns
SARA makes use of AI4SD's jarvis-fetcher GitHub App, requiring users to grant access through their GitHub accounts. Note that Jarvis will perform actions on the user's repository, such as adding webhooks and downloading files, but these actions will be done under Jarvis's name. We made sure to request only the permissions needed for this project, following the principle of least privilege.
How to use
Firstly, clone the AI4SD repository.
The backend is deployed in the hero-alliance-feup-ds-24-25 Google Cloud project so there are no required actions.
To run the frontend, open the AI4SD project in the folder /x-men/ai4sd-analysers in a new VSCode window. Then, run npm install and press F5. This will launch a new window where the AI4SD VSCode extension will be active. In this window, open any repository with Python files. Then, press Ctrl+Shift+P, and search for the command AI4SD Analysers: Analyse Repository. Select SARA, and choose from the dropdown menu if you'd like your local, or remote repo files analyzed (currently not working!), and which model will analyse them. Finally, wait for the results to appear in a markdown file!
If you'd like to try out SARA from the general AI4SD VSCode Extension, open a new window on the /x-men folder. Then, navigate to /x-men/ai4sd-analysers and run npm install followed by npm run compile. Then, press F5. A new window will open where the AI4SD Analysers extension is accessible, as well as any other sub-extensions you followed these steps for. Press Ctrl+Shift+P, and follow the above steps!
Sprint Retrospective
Sprint 1
"What went well?"
In our vision, one of the team's strong points during Sprint 1 was the way the team communicated and maintained good collaboration. Another fulcrum was the code quality, following a good GitHub Flow (usage of PRs, approvals, usage of branches for features, etc.), which ensured all the code followed a solid structure, resilient to failures.
"What went poorly?"
One of the points that did not help the team during the sprint review was the focus on other AI4SD components and the "neglect" of SARA. Although the other components are fulcral to the whole AI4SD system, the team should continue improving SARA continuously. Another point that decreased the performance of the team during the sprint was the development of parallel projects that took effort from the team members, who weren't able to allocate the desired time to the project.
"What can we do to improve?"
For the next sprint, we should communicate more with other teams to ensure not-so-repetitive tasks and be more agile during the sprint, to ensure that some unexpected events upon the sprint are quickly discussed and the sprint planning (PBIs) is modified to take those changes into consideration.
"What ideas do we have?"
The team suggested the idea of weekly "sprint reviews" to gather feedback on the status of the PBIs in progress and, if needed, to adapt the Scrum Board for this sprint, addressing the needs of time, to ensure that: What we aim is feasible for what we can do.
Scrum Board
This is our scrum board status at the end of Sprint 1.
Sprint 2
"What went well?"
From our perspective, the team highlighted herself as proactive in resolving problems in the AI4SD system, always available to discuss ideas, and contributing to the progress of AI4SD as a whole. Another aspect that the team performed well was the flexibility of working in non-comfort areas. As some developers began working on new areas that they weren't familiar with, they managed to perform well in those adversities.
"What went poorly?"
Similar to Sprint 1, one key issue negatively impacted the team's performance during this sprint: the development of parallel projects that diverted effort from team members, preventing them from dedicating the desired time to the project. Another factor that hindered the team was miscommunication between teams. During the sprint, there were instances of disagreement about design choices for the front end, which should have been addressed earlier during the prototype development for the X-Men. This misalignment prevented the team from improving as quickly as we had hoped.
"What can we do to improve?"
For the next sprint, the team should focus on improving knowledge sharing with other teams. As a group contributing to common components of the AI4SD system, it is essential to support those who want to collaborate on or integrate with these components, such as Jarvis, for instance.
"What ideas do we have?"
We suggested more meetings across all teams to address the problem of miscommunication mentioned before.
Scrum Board
This is our scrum board status at the end of Sprint 2.
Sprint 3
"What went well?"
From our perspective, the team highlighted itself once more as proactive in resolving problems in the AI4SD system, always available to discuss ideas, and contributing to the progress of AI4SD as a whole, maintaining a good performance. Another area where the team performed well was fixing features that did not match the desired performance.
"What went poorly?"
Similar to Sprint 1 and 2, an issue that continued to impact the team negatively was the development of parallel projects, which took time for the team and did not make it possible to work as much as areas as wanted. Another factor that hindered the team was the dependency some elements had on an issue from a developer of another team, where they had been blocked for a pretty good part of the sprint. More constant communication would have helped on that matter too.
"What can we do to improve?"
We gathered and discussed that we should focus on a set of guidelines for PR acceptance. We should take care of more special cases such as: Code Structure, fault tolerance, or even specific situations that fit that PR.
"What ideas do we have?"
We have the idea of creating a "Focus Day" where all the team members gather together to work on the project, where all the "desired" work for the week should be done within that "desired time", and only go over that time if things get very critical.
Scrum Board
This is our scrum board status at the end of Sprint 3.