Outreachy TRSS Machine Learning Project: Deep AQAtik - adoptium/aqa-test-tools GitHub Wiki

README for TRSS Machine Learning Project

(Codename ‘Deep AQAtik’)

This idea builds on some prototype work contributed to the Adoptium/AQAvit project for matching fresh test failures with existing open/reported issues. This work can be broken down into manageable pieces. The first part is to find a best fitting ‘list of open issues’ that matches a test failure - take our existing code that is using a particular model and try the same approach but varying the model to measure which models gives the most accurate matches with existing issues. This can be fairly open-ended work, but we will also engage the larger community at the project for input as well as involve the students in additional learning and research as well as consultation with data scientists.

After some experimentation with this, and having used the contents of various repositories (through queries to Github API to search through issues), the second piece can be to extrapolate this approach to an extended search through other content sources.

Matching possible issues could be extended beyond looking in github/JBS, to look for reported issues in StackOverFlow etc.

The goal of Test Result Summary Service (TRSS) Machine Learning Project is to suggest possible related GitHub issues when a nightly/weekly test fails. It also serves as a preliminary trial to introduce MLOps into our system, as we consider other problems that we could potentially solve (as discussed in this "Learning About Deep Learning presentation").

  • Both the ML training and deployment projects are implemented using Python 3 with Anaconda.
  • ML model training project utilizes sklearn, tensorflow, numpy, pandas, and several other packages to process data and train the model.
  • ML deployment project deploys the model saved from ML training project with Python Flask.
  • TRSS code lives in this aqa-test-tools repository and communicates with ML deployment server through RESTful API, and the ML deployment server may need to be set up for internal and external TRSS separately.

It will be useful for you to familiarize yourself with some or all of the tools listed above.

Currently:

  • training data is collected manually, we need to change it into an automatic process with monitoring related GitHub repo issues real-time.
  • we process the training data with tf-idf (term frequency–inverse document frequency), the accuracy could be further improved with more advanced algorithms.

Participants

Interns: Asjad Khan and Avishree Khare were interns on this project in its initial phases as part of an Outreachy term in 2021. Much work remains and we will continue this project as resources and people become available.

Mentors: Lan Xia, Longyu Zhang, Sophia Guo and Shelley Lambert are mentors on this project.