Project 2 Report - gregnr/SoftwareEvolutionAnalysis GitHub Wiki
#Introduction:
The goal of this project is to come up with an interesting question related to the evolution of software projects, then analysing real code bases to answer it. We are building a tool, Unit Test ROI Analyzer, with which we can see the relation between number of bugs and volume of unit tests over time.Our tool provides a graphical representation for finer analysing of the results. For more accurate results user can include pull requests, filter issues by given label and by keyword search.
Video Demonstration available on YouTube
Question
Is it possible that a relation exists between the amount of work that goes into creating test cases and the number of bugs that a project experiences? How does this change over time?
#####Hypothesis##### The amount of bugs a project has will decrease as the volume of unit tests increase.
#####Importance of this in Software Evolution##### It could be of great interest to developers as understanding a relationship between test cases and issues would help to allocate the proper amount of resources to QA.
#Methodology Tools:
- Git
- Github (issue tracking API)
- Node.js
- Plotly
Steps:
- Identify code bases that have a significant history of unit tests and provide issue tracking. We will be using Github "Explore" for this.
- Develop a script that will, given a GitHub repository, the name of its main branch, and a path to the unit test folder, determine two data sets: Frequency of bugs and volume of unit tests over time. We will gather the frequency of bugs by recording the number of issues opened per unit time. We will gather the volume of unit tests by measuring the number of lines of code in all files within the test directory per unit time. The unit of time will be weekly. The script will also be provided a username and password for the user's GitHub account, to avoid the rate limiting in the GitHub API. A user will also be able to filter out pull requests from issues, as well as filter the issues by their tag and keywords in the title/body. This will allow for more fine-grained analysis of the repository.
- Graph the data using plotly and make it available to the user through a web browser interface.
- Observe to see if there's any relation.
In terms of development, the team followed an Agile structure, and detailed sprint information is available through our own issues and milestones of the GitHub project. We used a Pull Request workflow so that no code was committed to the master branch without having some sort of code review first. This helps to weed out silly mistakes, bugs and keeps everyone up to date on the current state of the project.
Instructions to run/compile our code:
These instructions are for various Linux distributions (Ubuntu and CentOS tested)
On command line:
Install Node.js and its package manager NPM. See these instructions.
Clone the repository:
git clone https://github.com/gregnr/SoftwareEvolutionAnalysis
Navigate into the repository's directory. Run the command:
npm install
This will download dependencies used by the program into a node_modules directory.
Run the program by entering:
node src/main.js
Follow the prompts on screen.
On Internet browser:
Navigate to http://24.108.142.216:3000/
Note: this is a private server and will not always be available, especially because it isn't even a static IP. You can always run the server version by executing:
node src/server.js
and then navigating to http://localhost:3000/
Data sources used in our experiment:
The main data sources that will be presented are 3 repositories from GitHub that we ran our tool on, Ansible, Bootstrap and jQuery. With this in mind, the tool was actually ran on many more repositories, usually with similar results. Our tool was specifically designed with re-useablility in mind - it should work for any repository on GitHub, but the results will not generally mean anything unless the repository has both a testing directory and tracks their issues on GitHub.
List of metrics used in our experiment:
The two main metrics used in this experiment are lines of code in test cases and the number of open issues. These are calculated on a weekly basis. Finally, we established a derived metric called Change in Test Volume as follows,
Change in Test Volume(x) = abs(f(x+1) - f(x)) for (x = 0 to n-1), where
n = max number of weeks
x = a week
f(x) = test volume for week x
These metrics have been chosen because we feel that the number of currently open issues is closely related to the number of bugs that exist in a piece of software. We also feel the amount of work being done on the test cases is loosely related to how many lines of code are being added or removed (we didn't bother to investigate the difference - an addition or removal is just considered a "change"). The Change in Test Volume metric helps shed insight into whether the test cases are currently being worked on, and how much they are being worked on.
Results:
Raw Results (Ansible) :
| Week | Issues | Test Volume | Change in Test Volume |
|---|---|---|---|
| 146 | 80 | 14813 | 74 |
| 147 | 62 | 14925 | 112 |
| 148 | 71 | 16035 | 1110 |
| 149 | 63 | 16258 | 223 |
| 150 | 30 | 17234 | 976 |
| 151 | 32 | 17269 | 35 |
| 152 | 42 | 17269 | 0 |
| 153 | 61 | 17278 | 9 |
| 154 | 51 | 18209 | 931 |
| 155 | 58 | 18252 | 43 |
| 156 | 34 | 18327 | 75 |
| 157 | 63 | 18381 | 54 |
| 158 | 73 | 22388 | 4007 |
| 159 | 45 | 22502 | 114 |
| 160 | 50 | 22574 | 72 |
| 161 | 55 | 22658 | 84 |
| 162 | 50 | 22855 | 197 |
| 163 | 52 | 23557 | 702 |
| 164 | 53 | 23566 | 9 |
| 165 | 49 | 23566 | 0 |
| 166 | 12 | 24011 | 445 |
Final Graphs:
Ansible:
Bootstrap:
jQuery:
Analysis:
In Project 1, the tool did not include the necessary adjustments to allow for filtering out pull requests. This caused pull requests to be counted as issues. In Project 1, the results lead to us making some solid conclusions about the relation between bugs and testing.
However, in Project 2, once the pull requests and extraneous (non-bug) issues had been filtered out of our results, the conclusions that can be drawn become much weaker. As more repositories were analysed, the relations that we predicted in Project 1 also became less and less relevant. The curves for different projects were not smooth and often had striking differences, something that does not lend well to drawing any large general conclusions. Although we felt that the metrics we defined would help us to answer the question, maybe they are truly two independent quantities.
Regardless of this weakening, we still can make a prediction: in general, testing does not harm a project. That means that creating test cases rarely increases the bugs in the project. How much testing reduces bugs in a project still remains unclear. For this, we would recommend that future projects work on further filtering out bugs from other issues. The results could also be stronger if the algorithm worked on a smaller time scale. Something more fine grained than a week, such as a day or hours, might smooth out the graph a bit and make drawing conclusions easier.
There are a wealth of projects on GitHub that both have testing and track issues in the same place - this type of analysis works great on these types of projects. We leave the challenge of finding ways to track the same metrics for projects that don't follow this mantra for future investigators.
All in all, the answer to our original question is somewhat inconclusive. Although testing does not harm a project, we have proved nothing about what the exact effects of testing are on a project's issues. We successfully measured our metrics as defined, but the conclusions that can be drawn about their relation, once graphed against each other, remains a mystery.
Project Management Information
Milestones and timelines:
1.Integrate bootstrap and stand up the static UI
2.Formalize API and document
(March 11 - March 18, 2015)
1.Filter Issues by labels, keywords and whether they are pull request
2.Hookup server side API to our tool
(March 18 - March 25, 2015)
1 Get client side JS requests working to hookup API to static UI
2.UI artifacts for integrating with the Labels API
3.Add API for displaying list of project labels
(March 25 - April 1, 2015)
1.Integration tasks, testing and verification (polishing of software)
2.Analyse graphs, prepare presentation/report
(April 1 - April 8, 2015)
1.Deploy project to server for public access (future)
(By April 21, 2015)
Roles of team members:
Greg Richardson and Jordan Heemskerk: Chief developers.
Parker Atkins: Development/Analysis.
Rabjot Aujla: Project Management.