(Also see the Randoop Manual.)
Clone the repository, using either of these commands:
git clone git@github.com:randoop/randoop.git git clone https://github.com/randoop/randoop.git
When working on a particular feature, create a new
branch and work there rather
than in the master
branch.
To install prerequisites, execute (as superuser) the RUN commands in files
named scripts/Dockerfile-OPERATINGSYSTEM-jdkany
.
The most common commands to build Randoop are
./gradlew build manual
./gradlew assemble
./gradlew shadowJar
Windows users: you do not need to include the "./
" when running Gradle.
When our instructions say to run ./gradlew build
, you will run gradlew build
instead.
Generated files are
placed in subdirectories named build
in the parent directory and in
each of the agent subdirectories.
Jar files are placed in directory build/libs
.
You can also break this process down by running separate tasks.
For the full list of tasks supported by the Randoop build script, type
./gradlew tasks
.
Particular tasks that may be useful during development include
./gradlew jar |
to build the main Randoop jar file (but not agent or fat jar files) |
./gradlew shadowJar |
to build the "user" jar including all library dependencies |
./gradlew check |
to run all of the tests |
./gradlew testReport |
to generate HTML reports from tests
(in build/reports/allTests ) |
./gradlew clean |
to remove the build subdirectory |
./gradlew compileJava |
to compile the main source set |
./gradlew compileTestJava |
to compile the test source set |
./gradlew compileCoveredTestJava |
to compile the coveredTest source set |
./gradlew manual |
to update the manual from the source Javadoc |
Compiling uses the concept of
source sets.
A source set is a grouping of source files that can have its own set of
dependencies and tasks.
For instance, in the Randoop build.gradle
there is the
coveredTest
and replacecallTest
source sets that
are separate from the test
sourceset because the test classes
must run using the covered-class
and replacecall
Java
agents.
The Gradle Java plugin normally has two source sets named main
and test
,
and the Randoop build script defines others to better structure the build process.
Gradle allows you to run multiple tasks. For instance, for a clean build,
type ./gradlew clean build
.
The tasks cleanSite
, publishSite
, and buildRelease
should not be run except while making a release, and so should never be run
within a branch other than master.
These tasks will delete and/or replace the GitHub pages site located in the gh-pages
branch of the repository, which is checked out in a directory randoop-branch-gh-pages
that is a sibling of randoop
If you happen to run these commands, you should use git to revert to the previous version. For instance, the command
git checkout -- .
in the gh-pages
branch
can be used to restore the files before they have been committed.
Randoop is built with Gradle.
The shell script named ./gradlew
and the Windows batch script
gradlew.bat
run the Gradle wrapper for the project, which is the
recommended way to run Gradle commands.
Using the wrapper means you don't have to install Gradle yourself, instead
the wrapper will download Gradle the first time you run a build command.
To learn more about Gradle beyond the basic commands given here, see the Gradle command-line tutorial.
The Gradle build tasks for Randoop are defined in the build script build.gradle
,
along with the build scripts for the java agents in the corresponding subproject directories.
This is the file you would modify to change the Randoop build.
Note that the script uses plugins that each define tasks that may not be explicit
within the script, though it may be possible to configure them differently by
following the instructions at the plugin sites.
If you add or change a build task in a way that affects how Randoop is built, you should also update this document, which appears in the gh-pages branch. When you push your changes to the master branch on GitHub, you should also push to the gh-pages branch.
Randoop contains a number of unpublicized command-line options (because
they are experimental or unsupported). They are annotated with
the @Unpublicized
annotation in the source code.
When running Randoop, you should use the file
randoop-all-X.Y.Z.jar
that includes all dependencies.
For more detail, see the Randoop Manual.
The jar files for Java agents are located in
the build/libs
subdirectory of both the Randoop project and the
agent project directories.
So, for instance, the following will work to run the covered-class
agent:
-javaagent:<randoop-path>/build/libs/covered-class-4.3.3.jar
To run the Randoop tests, use the standard Gradle check
task:
./gradlew check
You can run a subset of the tests by using the --tests
option on the command line.
For instance,
./gradlew test --tests="*GenericTypesTest"
# You can include "run" in the test name, but still use a leading "*"
./gradlew systemTest --tests="*Collection*"
The system tests write logs to the directory
build/working-directories/test-name
.
You can create additional logs (as produced by
--selection-log
or
--operation-history-log
)
by setting the system properties
randoop.selection.log
or
randoop.operation.history.log
on the Gradle command line.
For instance:
./gradlew systemTest --tests="*Collection*" \
-Drandoop.log=randoop-log.txt -Drandoop.selection.log=selection-log.txt -Drandoop.operation.history.log=operation-log.txt
will create the files
build/working-directories/test-name/randoop-log.txt
build/selection-log.txt
build/operation-log.txt
Note that this works for system tests only, not for unit tests or other types of
tests.
Further note that selection logging can slow down the system tests so much that
they fail.
Finally, it should be noted that even though the default replacements attempt to suppress calls to
methods that use a graphical windowing system, not all such calls are captured.
Currently, the subtest runDirectSwingTest
will fail with java.awt.AWTError:
Local GraphicsEnvironment must not be null
if the system tests are run in a headless environment.
If Randoop generates an unexpected test (for example, Randoop's test suite outputs
java.lang.AssertionError: Test suite should have no error tests, but has 1:
),
you can run the test with commands like these:
javac -cp .:junit-4.12.jar -sourcepath .:.../randoop/src/testInput/java/ ErrorTest.java java -cp .:junit-4.12.jar:.../randoop/build/classes/java/testInput/ org.junit.runner.JUnitCore ErrorTestYou can minimize the test with a command like this:
java -ea -cp randoop-all-4.3.3.jar randoop.main.Main minimize \ --suitepath=ErrorTest0.java --suiteclasspath=.:junit-4.12.jar:.../randoop/classes/java/testInput/
Before making a release, and to evaluate the effect of changes to Randoop, you should compute coverage of Randoop-generated tests for 4 codebases.
build/reports/jacoco/test/html
and its subdirectories.
A summary of the results is in the file build/reports/jacoco/test/jacocoTestReport.xml
.
The overall coverage can be viewed by running the perl script: scripts/show-coverage.pl
.
This script will accept an optional argument of an alternative file location.
Invoke the script with -help
for a full list of options.
If you are planning to create or update a Randoop coverage data spreadsheet (see below)
you will need to run the perl script: scripts/prepare-coverage.pl
.
This tool reads the Jacoco coverage data and converts it into into a csv file named
"report-<input file modification date>.csv" in the current directory.
This script will accept an optional argument of an alternative file location.
Invoke the script with -help
for a full list of options.
After collecting the four sets of coverage data you will probably want to update an existing coverage spreadsheet or create a new one. The tools needed and the current spreadsheets are located in the coverage-tools repository. See the file README.md for details.
Azure Pipelines runs the Randoop continuous integration tests.
Each CI test runs a script in randoop/scripts/
, and they
currently pass on the master branch if this icon is green:
.
A standard practice is to create a branch and push it to the Randoop repository or to your own fork. CI will test it; if it succeeds, you can merge it. If it fails, you will be notified by email, and you can fix it without having destabilized the master branch.
The CI tests run in a Docker container. If you are not able to reproduce a CI test failure, then run the tests under Docker. To do so, first start a Docker container:
docker pull mdernst/randoop-ubuntu-jdk17 docker run -it mdernst/randoop-ubuntu-jdk17 /bin/bash
Then, in the Docker container, clone Randoop and run the appropriate script from randoop/scripts/
.
git clone --quiet -b BRANCHNAME --depth 9 https://github.com/USER/randoop.git randoop-fork-USER-branch-BRANCHNAME cd randoop-fork-USER-branch-BRANCHNAME ./scripts/test-nonSystemTest.sh
As you add new functionality to Randoop, you should also add new tests.
Randoop has three kinds of tests:
tests of Randoop internal classes (under directory src/test
),
tests of Randoop Java agents (under directories src/coveredTest
and src/replacecallTest
), and
system tests that run Randoop on input classes.
All of the tests are written as JUnit classes, but are separate because each
needs to be executed differently.
If you write JUnit tests that change the static state of either a command-line
argument or a static class, then you should add BeforeClass
and AfterClass
annotated methods that save and restore the prior state.
For command-line arguments, this can be done by adding the following code to
the beginning of the test class:
private static OptionsCache optionsCache;
@BeforeClass
public static void setup() {
optionsCache = new OptionsCache();
optionsCache.saveState();
}
@AfterClass
public static void restore() {
optionsCache.restoreState();
}
This uses the randoop.main.OptionsCache
class to save the static state of
all classes that include the definition of command-line arguments.
For classes under test with static state that is changed, you would use the
randoop.reflection.StaticCache
class to save and restore the state
of that particular class in a similar way.
The system tests are test methods of class randoop.main.RandoopSystemTest
in the src/systemTest
directory.
Each test uses one or more input classes from the testInput source set,
which resides in the src/testInput/java
subdirectory.
(System tests are run with a different classpath than the other tests, and
will only take input classes from src/testInput
.)
So, to set up a new system test, you need to add any new input classes, and add
a new test method to the system test class.
Here is an example test from randoop.main.RandoopSystemTest
that
illustrates the main themes that occur in writing a system test.
@Test
public void runLiteralsTest() {
// 1. Set up a working directory -- choose a unique name
TestEnvironment testEnvironment =
systemTestEnvironment.createTestEnvironment("literals-test");
// 2. Set options for test generation
RandoopOptions options = RandoopOptions.createOptions(testEnvironment);
options.setPackageName("");
options.setRegressionBasename("LiteralsReg");
options.setErrorBasename("LiteralsErr");
options.setOption("inputlimit", "1000");
options.addTestClass("randoop.literals.A");
options.addTestClass("randoop.literals.A2");
options.addTestClass("randoop.literals.B");
options.setOption("literals-level", "CLASS");
options.setOption("literals-file", "resources/systemTest/literalsfile.txt");
// 3. Indicate whether you expect regression tests (if you don't care use ExpectedTests.DONT_CARE)
ExpectedTests expectedRegressionTests = ExpectedTests.SOME;
ExpectedTests expectedErrorTests = ExpectedTests.NONE;
// 4. Run Randoop, compile and run generated tests, and check to see if expectations met
generateAndTest(testEnvironment, options, expectedRegressionTests, expectedErrorTests);
}
The options list is exactly what you would give at the command line, except for
setting the names and package for generated tests.
All Randoop options may be given, except for --junit-reflection-allowed=false
since the test run method only looks for JUnit suite classes.
Most existing tests take this form, using the RandoopSystemTest.generateAndTest()
method,
which checks whether Randoop generates the expected tests, that the tests compile,
and that regression tests pass and error tests fail when run.
The method also allows indicating for both kinds of tests, whether there should
be some (at least one) test, none, or if it doesn't matter.
This is done using the ExpectedTests
enumerated type, which has
values SOME
, NONE
, and DONT_CARE
.
However, it is possible to build a test differently using the utility methods of
the RandoopSystemTest
class.
At a minimum, a system test always attempts to compile generated tests.
It should also try to run any generated tests and confirm that regression tests
pass, and that error tests fail.
Additional information can be found in the RandoopSystemTest
class.
The build script creates files for use by IntelliJ IDEA, NetBeans, and Eclipse.
./gradlew ideaat the command line in the Randoop directory. This uses the IntelliJ IDEA plugin to create IDEA configuration files that IntelliJ uses when importing the project. You can then import the project as a Gradle project in IntelliJ IDEA, following any suggestions made on configuring Gradle support. See the Gradle tool window documentation at the IDEA Help site on how to work with Gradle projects.
./gradlew ideaat the command line in the Randoop directory. Then install the NetBeans plugin in NetBeans. This plugin uses the configuration files generated for the IDEA plugin during the import into NetBeans.
./gradlew eclipseat the command line in the Randoop directory. This runs the Gradle Eclipse plugin to configure the project for use with Eclipse. This will allow you to import Randoop as a project within Eclipse.
Other than .gitignore
, the tool-specific settings are not part of the
repository.
If your tool creates new configuration files, please add them to the
.gitignore
file.
If you edit build.gradle
,
please verify that it doesn't change behavior
from the command line using the Gradle wrapper.
Any user-visible changes should be documented in the changelog
src/docs/CHANGELOG.md
.
This text is used for documenting releases, so keeping it up-to-date and
ready for public release saves time in the long run.
Order the information by importance to the user.
Be sure to describe the change in terms of how it affects the user.
List all fixed bugs, which you can obtain by
querying
GitHub. There is no need to describe each one individually unless
users need to know about them individually.
Optionally, mention any serious reported issues that are not fixed.
The Randoop source code is formatted in Google Java Format by having Spotless run the google-java-format tool. The Randoop Gradle build script is setup to allow this tool to be applied using the following commands:
./gradlew spotlessCheck
to check formatting (does not work on Java 8)../gradlew spotlessApply
to reformat
all files in the repository.
Periodically, follow the instructions in
lib/README
to update local libraries.
(Background: Gradle downloads the appropriate version of
most of the libraries that Randoop uses.
However, Randoop also uses other local libraries that are kept in the lib
directory.)
The documentation for Randoop is in the subdirectory src/docs/manual
:
src/docs/manual/index.html
has the user manual.
src/docs/manual/dev.html
is this developer manual.
The command ./gradlew manual
updates command-line argument
documentation in the user manual, and updates the table-of-contents in both
manuals (in the src/docs/manual
directory).
The Javadoc is generated by the command ./gradlew javadoc
,
which places the files into build/docs/api/
.
These files are published to the project website by the publishSite
task described below.
Additional documentation that is not published to the project website is located
in src/docs
. This includes the change log, diagrams,
and project ideas documents.
The website, including the manuals, can be found in src/docs
.
These files, along with the Javadoc API, are moved to the
gh-pages
branch by the publishSite
task.
Prerequisites:
randoop
clone is another clone named randoop-branch-gh-pages
that
has checked out the gh-pages
branch.
To make a release from the master branch, follow these steps to ensure there are no changes needed:
git pull
./gradlew clean assemble javadoc validateAPI manual validateSite buildThis takes around 30 minutes. To shortcut it if no dependencies were updated, omit
clean
and build
.
./gradlew htmlSanityCheck
src/docs/CHANGELOG.md
contains all user-visible changes in
git log v$OLDVERSION..(See item (1.) below on how to set OLDVERSION.) Have a colleague review the changelog entry.
git status && git log --branches --not --remotes
Once there are no more changes to make:
OLDVERSION=4.3.3 NEWVERSION=4.3.4 NEXTVERSION=4.3.5
preplace `echo $NEWVERSION | sed 's/\./\\\./g'` $NEXTVERSION build.gradle preplace `echo $OLDVERSION | sed 's/\./\\\./g'` $NEWVERSION build.gradle (cd src && preplace `echo $NEWVERSION | sed 's/\./\\\./g'` $NEXTVERSION) (cd src && preplace `echo $OLDVERSION | sed 's/\./\\\./g'` $NEWVERSION)Undo undesirable changes to version numbers in
src/docs/CHANGELOG.md
.src/docs/manual/dev.html
.src/docs/manual/index.html
;
double-check against docs/CHANGELOG.md
.build/utils/checklink/checklink-args.txt
.checklink-args.txt
.
./gradlew clean buildRelease
gh-pages
branch:
git commit . -m "Update version to $NEWVERSION" && git push && \ (cd ../randoop-branch-gh-pages && \ git add -A . && git commit . -m "Update version to $NEWVERSION" && git push)
./gradlew checklinkThis command creates file
checklink-log.txt
.
If there are any other errors, then fix them by updating URLs in
Randoop or by suppressing a warning in
file checklink-args.txt
, which was mentioned above.
If you make any changes to Randoop, then commit and push the
changes, then start over at the ./gradlew clean
buildRelease
step.
./gradlew clean
buildRelease
step.
git tag -a v$NEWVERSION -m "Randoop version $NEWVERSION" git push --tags
build/distributions
,
upload the jar files from build/libs
,
and, finally, click "publish release".
To publish jars to Maven repositories:
./gradlew publishLocalPublicationToMavenLocal
publishes the artifacts to the local Maven repository.
./gradlew publishRemotePublicationToMavenRepository -PSONATYPE_NEXUS_USERNAME=username -PSONATYPE_NEXUS_PASSWORD=password
publishes the artifacts to the Nexus repository. If the version number ends in SNAPSHOT
, the release is published as a snapshot and no signing is required.
-Psigning.secretKeyRingFile=file -Psigning.password=secret -Psigning.keyId=id
to the command.
This section describes Randoop's main concepts, data structures and internal algorithms.
A unit test is a snippet of code that checks some expected property of some classes under test. It is helpful to think of a unit test as consisting of two things:
As a simple example, suppose we are testing the JDK's collections
classes (LinkedList
, Collections
, etc.) The
following unit test checks that the size of a linked list is computed
correctly after an element is added.
1. // Tests that the size of a list is 1 after adding an element. 2. public void testAdd() { 3. LinkedList<String> list = new LinkedList<>(); 4. boolean b = list.add(null); 5. assertEquals(1, list.size()); 6. }
The test input consists of lines 3-4, which exercises the code of the list class
by creating a list, and then calling the add
method.
The test check is on line 5.
In Randoop, a test input is represented as a Sequence, and test checks are represented using Checks. Each sequence is made up of one or more Statement objects.
Randoop's generation algorithms creates Sequences, adding the appropriate Checks to each, and output the results to the user as unit tests.
Randoop generates Sequence objects, which are used to construct the inputs of unit tests by adding checks. In Randoop, all test inputs are sequences of operations that use the methods and constructors of the classes under test.
Continuing with our running example, here is an example of a sequence of statements that calls methods from the classes under test: separate line:
LinkedList<String> l = new LinkedList<>(); String str = "hi!" l.addFirst(str); int i = l.size(); TreeSet<String> t = new TreeSet<>(l); Set<String> s = Collections.synchronizedSet(t);
Notice that each statement has three elements:
l
,
str
,
i
,
t
, and
s
).Rearranging the above sequence, we can see these elements more clearly:
Result Operation Inputs ====== ============= ====== statement 0: l = new LinkedList () statement 1: str = "hi!" () statement 2: addFirst (l, str) statement 3: i = size (l) statement 4: t = new TreeSet (l) statement 5: s = synchronizedSet (t)The three elements of a statement map to the following classes in Randoop:
A Operation represents the kind of operation that the statement performs.
Randoop operations include:
Method
class, and adds additional
functionality useful to Randoop.
null
value (in code, int x =
1
, String s = "s"
, Foo f = null
,
etc.).
int[] x = new int[] { var0, var1
};
).
s
, the variable s.getVariable(i)
represents the value produced by the i-th statement in the sequence.
You can create sequences in a variety of ways. The first way is via extension: take a sequence and add a new statement at the bottom. Recall the example sequence:
LinkedList<String> l = new LinkedList<>(); String str = "hi!"; l.addFirst(str); int i = l.size(); TreeSet<String> t = new TreeSet<>(l); Set s = Collections.synchronizedSet(t);
To create this sequence by extension we need the operations.
Because this code involves parameterized types, we need to instantiate the generic
class types that are used in order to get the type substitution that we need.
For instance, for the LinkedList<String>
constructor call, we
write:
InstantiatedType linkedListType = JDKTypes.LINKED_LIST_TYPE.instantiate(ConcreteTypes.STRING_TYPE); Substitution<ReferenceType> substLL = linkedListType.getTypeSubstitution(); TypedOperation newLL = TypedOperation.forConstructor(LinkedList.class.getConstructor()).substitute(substLL);
which first creates the parameterized type, gets the type substitution, and then applies the substitution to the operation created from the constructor of the generic class. The next operations aren't from generic classes, and don't require this amount of setup:
TypedOperation newOb = TypedOperation.createPrimitiveInitialization(ConcreteTypes.STRING_TYPE, "hi!"); TypedOperation addFirst = TypedOperation.forMethod(LinkedList.class.getMethod("addFirst", Object.class)).substitute(substLL); TypedOperation size = TypedOperation.forMethod(LinkedList.class.getMethod("size")).substitute(substLL);
But operations with wildcards require a substitution be applied first for the variables in the type, a capture conversion be applied, and then a substitution for the variables from the capture conversion.
InstantiatedType treeSetType = JDKTypes.TREE_SET_TYPE.instantiate(ConcreteTypes.STRING_TYPE); Substitution<ReferenceType> substTS = treeSetType.getTypeSubstitution(); TypedOperation wcTS = TypedOperation.forConstructor(TreeSet.class.getConstructor(Collection.class)).substitute(substTS).applyCaptureConversion(); Substitution<ReferenceType> substWC = new Substitution(wcTS.getTypeParameters(), (ReferenceType)ConcreteTypes.STRING_TYPE); TypedOperation newTS = wcTS.substitute(substWC);
And, similarly a generic operation requires building a substitution for the parameter to the method:
TypedOperation syncA = TypedOperation.forMethod(Collections.class.getMethod("synchronizedSet", Set.class)); Substitution<ReferenceType> substA = new Substitution(syncA.getTypeParameters(), (ReferenceType)ConcreteTypes.STRING_TYPE); TypedOperation syncS = syncA.substitute(substA);
Once the operations are defined with the proper types, we can build the sequence by extension:
Sequence s = new Sequence(); s = s.extend(newLL); s = s.extend(newOb); s = s.extend(addFirst, s.getVariable(0), s.getVariable(1)); s = s.extend(size, s.getVariable(0)); s = s.extend(newTS, s.getVariable(0)); s = s.extend(syncS, s.getVariable(4));
A couple notable points:
extend
operation
returns a new sequence rather than modifying its receiver.
A second way to create a sequence is using concatenation
:
given sequences s1
, s2
, and s3
,
you can create a new sequence that is the concatenation of the three
sequences.
List<Sequence> seqs = new ArrayList<Sequence>(); seqs.add(s1); seqs.add(s2); seqs.add(s3); Sequence newSeq = Sequence.concatenate(seqs);
If you're wondering why concatenation is useful, it is actually how
Randoop combines sequences to create new ones. To create a new
sequence that tests method m(A a, B b)
, Randoop first
finds and concatenates previously-generated sequences that create
objects of type A
and B
, and then extends
the concatenated sequence with a call of m
. See
Section
2.2. of this paper for more details.
A third way to create a sequence is by parsing it from a String. For
example, given a string parseable
with the following contents:
var0 = cons : java.util.LinkedList.<init>() : var1 = cons : java.lang.Object.<init>() : var2 = method : java.util.LinkedList.addFirst(java.lang.Object) : var0 var1 var3 = method : java.util.LinkedList.size() : var0 var4 = cons : java.util.TreeSet.<init>(java.util.Collection) : var0 var5 = method : java.util.Collections.synchronizedSet(java.util.Set) : var4
The following call creates a sequence corresponding to our running-example sequence:
Sequence seq = Sequence.parse(parseable);
An ExecutableSequence
wraps a Sequence
and adds two pieces of
functionality:
ExecutableSequence
can be augmented with
checks of expected properties. A Check is
an object that represents some expected property of a sequence;
for a example, that a specific method call in the sequence returns
normally. When an executable sequence is executed, any checks that are
present in the sequence are checked at runtime, and the
passing/failing status of the checks is available for the client to
inspect.
ExecutableSequence
can be executed.
Randoop uses Java's reflection mechanism to call the methods and
constructors in the sequence, and uses the structure of the
underlying Sequence
to determine what inputs to pass to
them.
Suppose we have created a Sequence s
The following two
lines will execute the sequence:
ExecutableSequence es = new ExecutableSequence(s); es.execute(null);
After the execute
method returns, you can access the
runtime objects created during execution via the getResult(int
i)
method, which returns the result of executing
statement i
. For example, assuming s
refers
to our running-example sequence from above, the following code prints
out
1
, i.e. the return value of the call to
method LinkedList.size()
.
ExecutableSequence es = new ExecutableSequence(s); es.execute(null); // Assuming statement at index 3 returned normally, print the runtime value ExecutionOutcome resultAt3 = es.getResult(3); if (resultAt3 instanceof NormalExecution) { System.out.println(((NormalExecution)resultAt3).getRuntimeValue()); }
To dig more into execution results, explore the classes Execution and ExecutionOutcome.
ExecutableSequence.executeStatement
. This is
accomplished by changing stdout and stderr to a memory-based
PrintSteam and recording the results.
Writing a list of executed sequences as JUnit files:
List<ExecutableSequence> sequences = ...; JunitFileWriter jfw = new JunitFileWriter(output_dir, junit_package_name, junit_classname, testsperfile); jfw.createJunitFiles(seq);
If you want to modify Randoop's JUnit-generating code, here are places to look:
randoop.JunitFileWriter
figures out how many JUnit classes/files
to write, how many tests to put in each, what to name them, etc.
randoop.JunitFileWriter.writeDriverFile
generates the non-reflective driver filerandoop.JunitFileWriter.writeSuiteFile
generates the (reflective) test suiterandoop.ExecutableSequence.toCodeString()
,randoop.ObjectContract
,randoop.Check
,Operation.appendCode(...)
A Check is an object that represents an expected property of a sequence. In terms of unit test concepts, a check represents some piece of checking code (or “test oracle”) of a unit test.
For example, consider the following unit test:
1. // Tests that the size of a list is 1 after adding an element. 2. @Test 3. public void testAdd() { 4. LinkedList l = new LinkedList(); // Create a list. 5. assertEquals(l, l); // List should be equal to itself. 6. Object o = new Object(); 7. boolean b = l.add(o); // Add an element to it. 8. org.junit.Assert.assertEquals(1, l.size()); // List should have size 1. 9. org.junit.Assert.assertEquals(l, l); // List should still be equal to itself. 10. int i = 10; 11. try { 12. Object o2 = l.remove(i); // Removing from invalid index 13. org.junit.Assert.fail(); // should throw exception. 14. } catch (IndexOutOfBoundsException e) { 15. // expected exception. 16. } 17. }
Like a typical unit test, this test combines test input code
(the code creating a list, adding an element to it, removing an
element, etc.) with checking code (the code in lines 6-7
creating assertions, and the code in lines 9, 11-14 checking that the
correct exception is thrown). In Randoop, test input code is
represented as a Sequence and checking code is
represented by Check
s associated with the sequence.
More specifically, each check is associated with an index in the sequence, where the index represents the time at which the check should be performed. For example, rearranging the above test to better see its structure, we have:
Checks performed Statement after statement ========= =============== l = new LinkedList() l.equals(l) returns true. o = new Object() no checks. b = l.add(o) l.size() returns 1. l.equals(l) returns true. i = 10; no checks. o2 = l.remove(i) throws IndexOutOfBoundsException.
The example illustrates how Randoop represents a test input along with its correctness checks.
Looking at the above example, you may reasonably ask: why do we draw a
distinction between “test input code” (the statements on the left) and
“checking code” (the statements on the right)? After all, aren't the
calls to size
and equals
in the checking
column also calls of the classes under test, and why not consider them
part of the test input?
We offer an answer in terms of how Randoop works. Randoop generates test inputs (Sequences) randomly, by combining and extending previously-generated sequences. On the other hand, Randoop performs its checks deterministically. For every sequence it creates, it performs checks on all the objects of the sequence.
Given how Randoop treats sequences and checks differently, it makes
more sense for a call like l.equals(l)
to be expressed as
a Check, so that it is always
(not randomly) performed.
(Under construction)
handle
is the
main GenTests
entrypoint for Randoop. (This is not strictly true, as Randoop's true
entrypoint is
class randoop.main.Main.
But GenTests
is where all the action starts with test
generation.) The handle
method is long and mostly deals with
setting up things before the generation process, and doing things like
outputting tests after generation.
ForwardGenerator
is the generator for Randoop's normal operation.