Projects used for evaluation

The following projects have been used for our evaluations with the EzUnit framework.

Project Version Source Remarks Matrices
Eclipse Draw2d 3.4.2 Source, Tests SVN Patch to apply; use "AllTests.java" TCMs
Eventbus 1.4 Link revision 95, SVN Patch to apply; exclude "TestPerformance.java" TCMs
Jester 1.37b Link latest version (Feb 12, 2012) TCMs
JExel 1.0.0b13 Link revision f504d9dc2474 (Oct 7, 2011) TCMs
JParsec 2.0.1 Link   TCMs
Jaxen 1.1.5 Link revision 1394 TCMs
HTML Parser 1.6 CVS: anonymous@htmlparser.cvs.sourceforge.net:/cvsroot/htmlparser data: May 06, 2009, SVN Patch to apply TCMs
Apache Commons Codec 1.3 Link revision 582448, SVN Patch to apply, use "AllTests.java" TCMs
Apache Commons Lang 3.0 Link revision 978864, SVN Patch to apply TCMs
Daikon 4.3.2 Link   TCMs

Archive format

For each project, an archive can be downloaded which contains the TCM for the initial run without any faults, and several TCMs for program versions with injected faults (use 7zip for unpacking). The structure of each archive is as follows:


|-- reference.txt
|-- 1-fault
    |-- [filename1].txt
    |-- [filename1].txt
    ...
|-- 2-fault
    |-- [filename1].txt
    ...
...

The file reference.txt contains the TCM of the unaltered program. Each subfolder contains TCMs for program versions with as many injected faults as denoted by the folder name. The file names contain the name of the projected and a timestamp when the run has been executed.

File format

Each TCM file consists of three sections. Each section begins with a '#' and a short description of the line format of that section (in EBNF), followed by the data sets, one set per line. Sections are divided by an empty line. The first section contains all tests executed by the test run with information about their result. The second section contains all executed units (e.g. methods); if a fault has been injected into the unit, the corresponding line will contain information about that fault. The third section contains the coverage matrix in column index format, where each line marks a test (in order of the tests in the first section) and contains the indices of the called units with the number of calls by that test. All indices are 0-indexed and point to the units in the second section.

An example file looks as follows:


  #tests: test_name, ' ', status(PASSED | FAILED | ERROR), ' ', [exception, { trace_mut_index | '(', { alternative_trace_mut_index, ',' }, ')' }]
  jester.tests.IgnoreListTest:testReadingEmptyIgnoreValues()V PASSED
  jester.tests.IgnoreListTest:testReadingIgnoreValues()V PASSED
  jester.tests.IgnoreListTest:testReadingIgnoreValuesEndOfLineSpecialCase()V PASSED
  jester.tests.IgnoreListTest:testReadingMultipleIgnoreValues()V PASSED
  jester.tests.IgnoreListTest:testReadingMultipleIgnoreBlankLines()V PASSED
  [...]

  #uuts: uut_name, [ '|', mutator_class, '|', change_position_in_file_buffer, '|', change ]
  jester.SourceChangeException:SourceChangeException(Ljava.lang.String;)V
  jester.RealMutationsList:RealMutationsList(Ljava.lang.String;Ljava.io.PrintStream;)V
  jester.RealClassTestTester:RealClassTestTester(Ljester.TestRunner;Ljester.MutationsList;)V
  jester.RealReport:redBar()V
  jester.ReportItem:ReportItem(Ljava.lang.String;Ljester.IgnoreListDocument;ILjava.lang.String;Ljava.lang.String;)V
  [...]

  #matrix (1 line per test, in order of tests): { mut_index, ' ', mut_call_count, ' ' }
  73 1 123 1 
  60 1 85 1 73 1 123 1 11 2 48 1 
  60 1 85 1 73 1 123 1 11 2 48 1 
  60 1 85 1 73 1 123 1 11 2 48 1 
  60 1 85 1 73 1 123 1 11 2 48 1 
  0 1 73 1 123 1 88 1 
  [...]

Readers for the matrix format have already been implemented for C++ and Java. The following sample programs can be used to read a given TCM for further evaluation.

C++ Download
Java Download