Projects with injected bugs - ligurio/sqa-wiki GitHub Wiki

The challenge binaries, valid test inputs, and sample vulnerabilities create an industry standard benchmark suite for evaluating:

  • Bug-finding tools
  • Program-analysis tools (e.g. automated test coverage generation, value range analysis)
  • Patching strategies
  • Exploit mitigations
  • Learning software testing techniques

Software projects with injected defects:

  • C/C++ FuzzBench https://google.github.io/fuzzbench/
  • C/C++ Memory sanitizer benchmark
  • C/C++, Java Juliet Test Suite
  • Java droixbench is a benchmark that contains 24 reproducible crashes in open source Android apps.
  • Java Defects4J is a benchmark of 341 Java bugs from 5 open-source projects. It contains the corresponding patches, which cover a variety of patch type.
  • Java Bears is a benchmark of continuous integration build failures focusing on test failures.
  • Java Bugs.jar — a large-scale, diverse dataset of bugs for Java program repair.
  • Java REST service (https://martinfowler.com/articles/practical-test-pyramid.html)
  • Java Juliet Test Suite for Java
  • Java tarpit-java - a web application seeded with vulnerabilities, rootkits, backdoors and data leaks.
  • Java Web EasyBuggy is a broken web application in order to understand behavior of bugs and vulnerabilities, for example, memory leak, deadlock, JVM crash, SQL injection and so on.
  • Python tarpit-python - a web application seeded with vulnerabilities, rootkits, backdoors and data leaks.
  • C C++ IntroClass — automated program repair benchmark that consists of 998 defects in small student-written programming assignments.
  • C C++ DBGBench — 291 (in)correct patches from real software professionals for 27 real bugs in C for the qualitative evaluation of automated repair techniques.
  • C C++ Codeflaws — 3902 bugs from Codeforces programming competition for evaluating program repair tools across different defect classes.
  • C Space
  • C "Siemens" programs were assembled by Tom Ostrand and colleagues at Siemens Corporate Research for a study of the fault detection capabilities of control-flow and data-flow coverage criteria [Hutchins94], and were made available to us by Tom Ostrand.
  • C C++ ManyBugs is a benchmark of 185 C bugs in nine open-source programs.
  • C tarpit-c - a set of C/C++ code snippets seeded with vulnerable conditions.
  • C Program Bug Examples
  • Web ParkCalc - Agile Testing Challenge
  • Multilingual BugSwarm — a dataset of thousands of real software bugs and their fixes.
  • Multilingual QuixBugs — a parallel corpus of 40 programs in both Python and Java, each with a bug on one line.
  • DARPA Challenge Binaries are custom-made programs specifically designed to contain vulnerabilities that represent a wide variety of crashing software flaws.
  • DARPA Cyber Grand Challenge Sample Challenges - What is the Cyber Grand Challenge?