Package randoop.output

Class FailingAssertionCommentWriter

java.lang.Object

randoop.output.FailingAssertionCommentWriter

All Implemented Interfaces:

CodeWriter

public class FailingAssertionCommentWriter
extends Object
implements CodeWriter

A CodeWriter that comments out failing assertions when outputting JUnit tests. This disables flaky tests that pass when run reflectively within Randoop, but fail when run from the command line. The whole test is not commented or removed, because the test might have side effects that later tests depend upon.

Iteratively does the following:

• Writes the class.
• Compiles and runs the tests in a new JVM to determine whether there are failing assertions.
• Replaces each failing assertion by a comment containing the code for the failing assertion.

Creates a clean temporary directory for each compilation/run of a test class to avoid state effects due to files in the working directory.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	FailingAssertionCommentWriter.Match	The line and first group from the match of a Pattern.
private static class	FailingAssertionCommentWriter.NotMatchedException	An exception that indicates that an expected pattern was not found.

Field Summary

Fields

Modifier and Type	Field	Description
private static Pattern	FAILURE_HEADER_PATTERN
private static Pattern	FAILURE_MESSAGE_PATTERN	A pattern matching the JUnit4 message indicating the total count of failures.
private HashSet<String>	flakyTestNames	Method names for flaky tests (e.g., "test005").
private JavaFileWriter	javaFileWriter	The underlying JavaFileWriter for writing a test class.
private TestEnvironment	testEnvironment	The TestEnvironment for running the test classes.
private static String	TYPE_REGEX	Matches a type: a class name, optional generics, optional array brackets.
private static Pattern	VARIABLE_DECLARATION_LINE	Matches a variable declaration.

Constructor Summary

Constructors

Constructor	Description
FailingAssertionCommentWriter(TestEnvironment testEnvironment, JavaFileWriter javaFileWriter)	Create a FailingAssertionCommentWriter.

Method Summary

Modifier and Type	Method	Description
private String	commentCatchStatements(String packageName, String javaCode, List<Diagnostic<? extends JavaFileObject>> diagnostics, Path destinationDir, FileCompiler.FileCompilerException e)	Comments out lines with unnecessary catch or try statements.
private String	commentFailingAssertions(String packageName, String classname, String javaCode, RunCommand.Status status, Set<String> flakyTests)	Comments out lines with failing assertions.
private void	compilationError(Path destinationDir, String classSource, List<Diagnostic<? extends JavaFileObject>> diagnostics, FileCompiler.FileCompilerException e)	Issue an exception because of a non-recoverable compilation error.
private Path	compileTestClass(String packageName, String classname, String classSource, Path destinationDir)	Compiles the Java files in the list of files and writes the resulting class files to the directory.
private Path	createWorkingDirectory(String classname, int pass)	Creates a temporary directory by concatenating the class name and a pass count to form the directory name.
private String	flakyLineReplacement(String flakyLine, String failure)	Given a flaky line (one that throws an exception but was not expected to), return a commented version of the line that does no computation.
Set<String>	getFlakyTestNames()	Returns the set of flaky test names.
private int	numJunitFailures(Iterator<String> lineIterator, RunCommand.Status status, String qualifiedClassname, String javaCode)	Return the number of JUnit failures, parsed from the JUnit output.
private FailingAssertionCommentWriter.Match	readUntilMatch(Iterator<String> lineIterator, Pattern pattern)	Reads lines of the JUnit output using the iterator until finding a match for the pattern, and then returns a pair containing the line and the text matching the first group of the pattern.
Path	writeClassCode(String packageName, String classname, String classSource)	Writes the given class using this CodeWriter.
Path	writeUnmodifiedClassCode(String packageName, String classname, String javaCode)	Writes the given class.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FAILURE_MESSAGE_PATTERN

private static final Pattern FAILURE_MESSAGE_PATTERN

A pattern matching the JUnit4 message indicating the total count of failures. Capturing group 1 is the number of failures.

FAILURE_HEADER_PATTERN

private static final Pattern FAILURE_HEADER_PATTERN

TYPE_REGEX

private static final String TYPE_REGEX

Matches a type: a class name, optional generics, optional array brackets.

See Also:

Constant Field Values

VARIABLE_DECLARATION_LINE

private static final Pattern VARIABLE_DECLARATION_LINE

Matches a variable declaration. Capturing group 1 is through the "=", 2 is the type, 3 is the initializer.

testEnvironment

private final TestEnvironment testEnvironment

The TestEnvironment for running the test classes.

javaFileWriter

private final JavaFileWriter javaFileWriter

The underlying JavaFileWriter for writing a test class.

flakyTestNames

private final HashSet<String> flakyTestNames

Method names for flaky tests (e.g., "test005").

Constructor Detail

FailingAssertionCommentWriter

public FailingAssertionCommentWriter(TestEnvironment testEnvironment,
                                     JavaFileWriter javaFileWriter)

Create a FailingAssertionCommentWriter.

Parameters:

testEnvironment - the TestEnvironment for executing tests during filtering

javaFileWriter - the JavaFileWriter to write .java files for the classes

Method Detail

getFlakyTestNames

public Set<String> getFlakyTestNames()

Returns the set of flaky test names. Each element has the form testNNN where N are digits; for example, "test002".

Returns:

the flaky test names

writeClassCode

public Path writeClassCode(String packageName,
                           String classname,
                           String classSource)
                    throws RandoopOutputException

Writes the given class using this CodeWriter. May modify the class text before writing it. May write additional files (but those are not returned).

Replaces failing assertions by comments.

Assumes output from JUnit4 org.junit.runner.JUnitCore runner used in TestEnvironment.

Specified by:

writeClassCode in interface CodeWriter

Parameters:

packageName - the package name of the class

classname - the simple name of the class

classSource - the text of a Java class, must be compilable

Returns:

the Path object for the Java file written

Throws:

RandoopOutputException - if there is an error while writing the code

writeUnmodifiedClassCode

public Path writeUnmodifiedClassCode(String packageName,
                                     String classname,
                                     String javaCode)
                              throws RandoopOutputException

Description copied from interface: CodeWriter

Writes the given class. Does not modify the class text.

Specified by:

writeUnmodifiedClassCode in interface CodeWriter

Parameters:

packageName - the package name of the class

classname - the simple name of the class

javaCode - the text of the class to be written, must be compilable

Returns:

the Path object for the Java file written

Throws:

RandoopOutputException - if there is an error while writing the code

commentCatchStatements

private String commentCatchStatements(String packageName,
                                      String javaCode,
                                      List<Diagnostic<? extends JavaFileObject>> diagnostics,
                                      Path destinationDir,
                                      FileCompiler.FileCompilerException e)

Comments out lines with unnecessary catch or try statements. Fails if any other compilation errors exist. Ignores compilation warnings.

Parameters:

packageName - the package name of the test class

javaCode - the source code for the test class; each assertion must be on its own line

diagnostics - the errors and warnings from compiling the class

destinationDir - the directory that contains the source code, used only for debugging

e - the exception that was raised when compiling the source code, used only for debugging

Returns:

the class source edited so that failing assertions are replaced by comments

Throws:

RandoopBug - if there is an unhandled compilation error (i.e., not about an unnecessary catch or try statement)

compilationError

private void compilationError(Path destinationDir,
                              String classSource,
                              List<Diagnostic<? extends JavaFileObject>> diagnostics,
                              FileCompiler.FileCompilerException e)

Issue an exception because of a non-recoverable compilation error.

Parameters:

destinationDir - the directory that contains the source code, used only for debugging

classSource - the text of the test class

diagnostics - the errors and warnings from compiling the class

e - the exception that was raised when compiling the source code, used only for debugging

commentFailingAssertions

private String commentFailingAssertions(String packageName,
                                        String classname,
                                        String javaCode,
                                        RunCommand.Status status,
                                        Set<String> flakyTests)

Comments out lines with failing assertions. Uses the failures in the status from running JUnit with javaCode to identify lines with failing assertions.

Parameters:

packageName - the package name of the test class

classname - the simple (unqualified) name of the test class

javaCode - the source code for the test class; each assertion must be on its own line

status - the result of running the test with JUnit

flakyTests - names of flaky tests, e.g. "test005". This is an output parameter that is augmented by this method.

Returns:

the class source edited so that failing assertions are replaced by comments

Throws:

RandoopBug - if status contains output for a failure not involving a Randoop-generated test method

numJunitFailures

private int numJunitFailures(Iterator<String> lineIterator,
                             RunCommand.Status status,
                             String qualifiedClassname,
                             String javaCode)

Return the number of JUnit failures, parsed from the JUnit output.

Parameters:

lineIterator - an iterator over the lines of JUnit output

status - the result of running JUnit

qualifiedClassname - the name of the JUnit class, used only for debugging output

javaCode - the JUnit class source code, used only for debugging output

Returns:

the number of JUnit failures, a non-negative integer

flakyLineReplacement

private String flakyLineReplacement(String flakyLine,
                                    String failure)

Given a flaky line (one that throws an exception but was not expected to), return a commented version of the line that does no computation.

Parameters:

flakyLine - the line that throws an exception

Returns:

the line, with its computation commented out

readUntilMatch

private FailingAssertionCommentWriter.Match readUntilMatch(Iterator<String> lineIterator,
                                                           Pattern pattern)

Reads lines of the JUnit output using the iterator until finding a match for the pattern, and then returns a pair containing the line and the text matching the first group of the pattern. Assumes that there is a match, and that the pattern has at least one group.

Parameters:

lineIterator - the iterator for reading from the JUnit output

pattern - the pattern for a regex with at least one group

Returns:

the pair containing the line and the text matching the first group

Throws:

RandoopBug - if the iterator has no more lines, but the pattern hasn't been matched

compileTestClass

private Path compileTestClass(String packageName,
                              String classname,
                              String classSource,
                              Path destinationDir)
                       throws FileCompiler.FileCompilerException

Compiles the Java files in the list of files and writes the resulting class files to the directory.

Parameters:

packageName - the package name for the test class

classname - the name of the test class

classSource - the text of the test class

destinationDir - the directory for class file output

Returns:

the name of the file

Throws:

FileCompiler.FileCompilerException - if the file does not compile

createWorkingDirectory

private Path createWorkingDirectory(String classname,
                                    int pass)

Creates a temporary directory by concatenating the class name and a pass count to form the directory name.

Parameters:

classname - the class name

pass - the pass count

Returns:

the Path for the directory created