Randoop logo

Randoop Developer's Manual

(Also see the Randoop Manual.)

Getting started

Clone the repository, using either of these commands:

  git clone git@github.com:randoop/randoop.git
  git clone https://github.com/randoop/randoop.git

When working on a particular feature, create a new branch and work there rather than in the master branch.

Prerequisites

To install prerequisites, execute (as superuser) the RUN commands in files named scripts/Dockerfile-OPERATINGSYSTEM-jdkany.

Building Randoop

In Randoop's top-level directory, run 

  ./gradlew build manual

This command compiles Randoop, compiles the Java agents, runs all tests, builds the jar files, and updates the manual from the source Javadoc. Generated files are placed in subdirectories named build in the parent directory and in each of the agent subdirectories. Jar files are placed in directory build/libs.

Windows users: you do not need to include the "./" when running Gradle. When our instructions say to run ./gradlew build, you will run gradlew build instead.

You can also break this process down by running separate tasks. For the full list of tasks supported by the Randoop build script, type ./gradlew tasks. Particular tasks that may be useful during development include

./gradlew jar to build the main Randoop jar file (but not agent or fat jar files)
./gradlew assemble to compile and build Randoop and fat jar files
./gradlew shadowJar to build the "user" jar including all library dependencies
./gradlew check to run all of the tests
./gradlew testReport to generate HTML reports from tests (in build/reports/allTests)
./gradlew clean to remove the build subdirectory
./gradlew compileJava to compile the main source set
./gradlew compileTestJava to compile the test source set
./gradlew compileCoveredTestJava to compile the coveredTest source set
./gradlew manual to update the manual from the source Javadoc

Compiling uses the concept of source sets. A source set is a grouping of source files that can have its own set of dependencies and tasks. For instance, in the Randoop build.gradle there is the coveredTest and replacecallTest source sets that are separate from the test sourceset because the test classes must run using the covered-class and replacecall Java agents. The Gradle Java plugin normally has two source sets named main and test, and the Randoop build script defines others to better structure the build process.

Gradle allows you to run multiple tasks. For instance, for a clean build, type ./gradlew clean build.

Release-only tasks

The tasks cleanSite, publishSite, and buildRelease should not be run except while making a release, and so should never be run within a branch other than master. These tasks will delete and/or replace the GitHub pages site located in the gh-pages branch of the repository, which is checked out in a directory randoop-branch-gh-pages that is a sibling of randoop

If you happen to run these commands, you should use git to revert to the previous version. For instance, the command 

  git checkout -- .

in the gh-pages branch can be used to restore the files before they have been committed.

The Gradle wrapper

Randoop is built with Gradle. The shell script named ./gradlew and the Windows batch script gradlew.bat run the Gradle wrapper for the project, which is the recommended way to run Gradle commands. Using the wrapper means you don't have to install Gradle yourself, instead the wrapper will download Gradle the first time you run a build command.

To learn more about Gradle beyond the basic commands given here, see the Gradle command-line tutorial.

The Randoop build script

The Gradle build tasks for Randoop are defined in the build script build.gradle, along with the build scripts for the java agents in the corresponding subproject directories. This is the file you would modify to change the Randoop build. Note that the script uses plugins that each define tasks that may not be explicit within the script, though it may be possible to configure them differently by following the instructions at the plugin sites.

If you add or change a build task in a way that affects how Randoop is built, you should also update this document, which appears in the gh-pages branch. When you push your changes to the master branch on GitHub, you should also push to the gh-pages branch.

Running Randoop

Randoop contains a number of unpublicized command-line options (because they are experimental or unsupported). They are annotated with the @Unpublicized annotation in the source code.

Randoop classpath

When running Randoop, you should use the file randoop-all-X.Y.Z.jar that includes all dependencies. For more detail, see the Randoop Manual.

The jar files for Java agents are located in the build/libs subdirectory of both the Randoop project and the agent project directories. So, for instance, the following will work to run the covered-class agent: 

-javaagent:<randoop-path>/build/libs/covered-class-4.3.2.jar

Testing Randoop

To run the Randoop tests, use the standard Gradle check task: 

./gradlew check

You can run a subset of the tests by using the --tests option on the command line. For instance, 

    ./gradlew test --tests="*GenericTypesTest"
    # You can include "run" in the test name, but still use a leading "*"
    ./gradlew systemTest --tests="*Collection*"

The system tests write logs to the directory build/working-directories/test-name. You can create additional logs (as produced by --selection-log or --operation-history-log) by setting the system properties randoop.selection.log or randoop.operation.history.log on the Gradle command line. For instance: 

  ./gradlew systemTest --tests="*Collection*" \
    -Drandoop.log=randoop-log.txt -Drandoop.selection.log=selection-log.txt -Drandoop.operation.history.log=operation-log.txt

will create the files 

  build/working-directories/test-name/randoop-log.txt
  build/selection-log.txt
  build/operation-log.txt

Note that this works for system tests only, not for unit tests or other types of tests. Further note that selection logging can slow down the system tests so much that they fail. Finally, it should be noted that even though the default replacements attempt to suppress calls to methods that use a graphical windowing system, not all such calls are captured. Currently, the subtest runDirectSwingTest will fail with java.awt.AWTError: Local GraphicsEnvironment must not be null if the system tests are run in a headless environment.

If Randoop generates an unexpected test (for example, Randoop's test suite outputs java.lang.AssertionError: Test suite should have no error tests, but has 1:), you can run the test with commands like these: 

javac -cp .:junit-4.12.jar -sourcepath .:.../randoop/src/testInput/java/ ErrorTest.java
java -cp .:junit-4.12.jar:.../randoop/build/classes/java/testInput/ org.junit.runner.JUnitCore ErrorTest
You can minimize the test with a command like this: 
java -ea -cp randoop-all-4.3.2.jar randoop.main.Main minimize \
  --suitepath=ErrorTest0.java --suiteclasspath=.:junit-4.12.jar:.../randoop/classes/java/testInput/

Checking Randoop code coverage

Before making a release, and to evaluate the effect of changes to Randoop, you should compute coverage of Randoop-generated tests for 4 codebases.

After collecting the four sets of coverage data you will probably want to update an existing coverage spreadsheet or create a new one. The tools needed and the current spreadsheets are located in the coverage-tools repository. See the file README.md for details.

Running tests under CI, and reproducing them outside CI

Azure Pipelines runs the Randoop continuous integration tests. Each CI test runs a script in randoop/scripts/, and they currently pass on the master branch if this icon is green: Randoop Azure Pipelines status icon.

A standard practice is to create a branch and push it to the Randoop repository or to your own fork. CI will test it; if it succeeds, you can merge it. If it fails, you will be notified by email, and you can fix it without having destabilized the master branch.

The CI tests run in a Docker container. If you are not able to reproduce a CI test failure, then run the tests under Docker. To do so, first start a Docker container: 

docker pull mdernst/randoop-ubuntu-jdk17
docker run -it mdernst/randoop-ubuntu-jdk17 /bin/bash

Then, in the Docker container, clone Randoop and run the appropriate script from randoop/scripts/. 

git clone --quiet -b BRANCHNAME --depth 9 https://github.com/USER/randoop.git randoop-fork-USER-branch-BRANCHNAME
cd randoop-fork-USER-branch-BRANCHNAME
./scripts/test-nonSystemTest.sh

Adding tests

As you add new functionality to Randoop, you should also add new tests. Randoop has three kinds of tests: tests of Randoop internal classes (under directory src/test), tests of Randoop Java agents (under directories src/coveredTest and src/replacecallTest), and system tests that run Randoop on input classes. All of the tests are written as JUnit classes, but are separate because each needs to be executed differently.

Unit tests

If you write JUnit tests that change the static state of either a command-line argument or a static class, then you should add BeforeClass and AfterClass annotated methods that save and restore the prior state. For command-line arguments, this can be done by adding the following code to the beginning of the test class: 


  private static OptionsCache optionsCache;

  @BeforeClass
  public static void setup() {
    optionsCache = new OptionsCache();
    optionsCache.saveState();
  }

  @AfterClass
  public static void restore() {
    optionsCache.restoreState();
  }

This uses the randoop.main.OptionsCache class to save the static state of all classes that include the definition of command-line arguments. For classes under test with static state that is changed, you would use the randoop.reflection.StaticCache class to save and restore the state of that particular class in a similar way.

System tests

The system tests are test methods of class randoop.main.RandoopSystemTest in the src/systemTest directory. Each test uses one or more input classes from the testInput source set, which resides in the src/testInput/java subdirectory. (System tests are run with a different classpath than the other tests, and will only take input classes from src/testInput.) So, to set up a new system test, you need to add any new input classes, and add a new test method to the system test class.

Here is an example test from randoop.main.RandoopSystemTest that illustrates the main themes that occur in writing a system test. 


 @Test
 public void runLiteralsTest() {

   // 1. Set up a working directory -- choose a unique name
  TestEnvironment testEnvironment =
        systemTestEnvironment.createTestEnvironment("literals-test");

   // 2. Set options for test generation
  RandoopOptions options = RandoopOptions.createOptions(testEnvironment);
  options.setPackageName("");
  options.setRegressionBasename("LiteralsReg");
  options.setErrorBasename("LiteralsErr");
  options.setOption("inputlimit", "1000");
  options.addTestClass("randoop.literals.A");
  options.addTestClass("randoop.literals.A2");
  options.addTestClass("randoop.literals.B");
  options.setOption("literals-level", "CLASS");
  options.setOption("literals-file", "resources/systemTest/literalsfile.txt");

  // 3. Indicate whether you expect regression tests (if you don't care use ExpectedTests.DONT_CARE)
  ExpectedTests expectedRegressionTests = ExpectedTests.SOME;
  ExpectedTests expectedErrorTests = ExpectedTests.NONE;

  // 4. Run Randoop, compile and run generated tests, and check to see if expectations met
  generateAndTest(testEnvironment, options, expectedRegressionTests, expectedErrorTests);
 }

The options list is exactly what you would give at the command line, except for setting the names and package for generated tests. All Randoop options may be given, except for --junit-reflection-allowed=false since the test run method only looks for JUnit suite classes.

Most existing tests take this form, using the RandoopSystemTest.generateAndTest() method, which checks whether Randoop generates the expected tests, that the tests compile, and that regression tests pass and error tests fail when run. The method also allows indicating for both kinds of tests, whether there should be some (at least one) test, none, or if it doesn't matter. This is done using the ExpectedTests enumerated type, which has values SOME, NONE, and DONT_CARE.

However, it is possible to build a test differently using the utility methods of the RandoopSystemTest class. At a minimum, a system test always attempts to compile generated tests. It should also try to run any generated tests and confirm that regression tests pass, and that error tests fail. Additional information can be found in the RandoopSystemTest class.

Editing Randoop

Using an IDE

The build script creates files for use by IntelliJ IDEA, NetBeans, and Eclipse.

Other than .gitignore, the tool-specific settings are not part of the repository. If your tool creates new configuration files, please add them to the .gitignore file. If you edit build.gradle, please verify that it doesn't change behavior from the command line using the Gradle wrapper.

Documenting changes

Any user-visible changes should be documented in the changelog src/docs/CHANGELOG.md. This text is used for documenting releases, so keeping it up-to-date and ready for public release saves time in the long run. Order the information by importance to the user. Be sure to describe the change in terms of how it affects the user. List all fixed bugs, which you can obtain by querying GitHub. There is no need to describe each one individually unless users need to know about them individually. Optionally, mention any serious reported issues that are not fixed.

Formatting Randoop source code

The Randoop source code is formatted in Google Java Format by having Spotless run the google-java-format tool. The Randoop Gradle build script is setup to allow this tool to be applied using the following commands:

Updating libraries

Periodically, follow the instructions in lib/README to update local libraries. (Background: Gradle downloads the appropriate version of most of the libraries that Randoop uses. However, Randoop also uses other local libraries that are kept in the lib directory.)

Modifying the manual

The documentation for Randoop is in the subdirectory src/docs/manual:

The command ./gradlew manual updates command-line argument documentation in the user manual, and updates the table-of-contents in both manuals (in the src/docs/manual directory).

The Javadoc is generated by the command ./gradlew javadoc, which places the files into build/docs/api/. These files are published to the project website by the publishSite task described below.

Additional documentation that is not published to the project website is located in src/docs. This includes the change log, diagrams, and project ideas documents.

Modifying the website

The website, including the manuals, can be found in src/docs. These files, along with the Javadoc API, are moved to the gh-pages branch by the publishSite task.

Releasing a new version of Randoop

Prerequisites:

  1. The release process assumes that a sibling of your randoop clone is another clone named randoop-branch-gh-pages that has checked out the gh-pages branch.
  2. The release can be made on any machine. The machine doesn't have to be connected to the CSE filesystem.

To make a release from the master branch, follow these steps to ensure there are no changes needed:

  1. Ensure that your clone is up to date: 
    git pull
  2. Compile, test, and build documentation. If there is any failure, fix them and start over. 
    ./gradlew clean assemble javadoc validateAPI manual validateSite build
    This takes around 30 minutes. To shortcut it if no dependencies were updated, omit clean and build. Note that validateAPI will not be run on Java 8.
  3. Run another HTML checker. Look at the HTML file it outputs (build/reports/htmlSanityCheck/index.html), not its textual output on standard out. It suffers 1 false positive report: "1 missing local resources found. ... local resource "api/" missing". If it produces any true reports, fix them and start over. 
    ./gradlew htmlSanityCheck
  4. Double-check that the changelog src/docs/CHANGELOG.md contains all user-visible changes in 
    git log v$OLDVERSION..
    (See item (1.) below on how to set OLDVERSION.) Have a colleague review the changelog entry.
  5. Commit any changes; push any unpushed commits. Ensure you have done so by running: 
    git status && git log --branches --not --remotes

Once there are no more changes to make:

  1. Increment the Randoop version number:
    1. Set environment variables for the old and new version numbers. (Don't necessarily use the examples given here!) 
      OLDVERSION=4.3.2
NEWVERSION=4.3.3
NEXTVERSION=4.3.4
    2. Update the version number by running the following commands. 
        preplace `echo $NEWVERSION | sed 's/\./\\\./g'` $NEXTVERSION build.gradle
  preplace `echo $OLDVERSION | sed 's/\./\\\./g'` $NEWVERSION build.gradle
  (cd src && preplace `echo $NEWVERSION | sed 's/\./\\\./g'` $NEXTVERSION)
  (cd src && preplace `echo $OLDVERSION | sed 's/\./\\\./g'` $NEWVERSION)
      Undo undesirable changes to version numbers in src/docs/CHANGELOG.md.
      Update OLDVERSION, NEWVERSION, and NEXTVERSION for next release in src/docs/manual/dev.html.
      Update the date in src/docs/manual/index.html; double-check against docs/CHANGELOG.md.
      Double-check all uncommitted changes.
      Stage the changes (using git add), but don't commit the changes yet.
    3. Change the version number for the three exceptions for the Randoop zip file in checklink-args.txt, for instance by editing local file build/utils/checklink/checklink-args.txt.
      Commit and push your changes to checklink-args.txt.
  2. Create the distribution files. This runs the tests again, so it takes about half an hour. 
      ./gradlew clean buildRelease
  3. Commit the changes and update the project site in the gh-pages branch: 
      git commit . -m "Update version to $NEWVERSION" && git push && \
  (cd ../randoop-branch-gh-pages && \
  git add -A . && git commit . -m "Update version to $NEWVERSION" && git push)
  4. Check that there are no broken links on the published website: 
    ./gradlew checklink
    This command creates file checklink-log.txt. If there are any other errors, then fix them by updating URLs in Randoop or by suppressing a warning in file checklink-args.txt, which was mentioned above. If you make any changes to Randoop, then commit and push the changes, then start over at the ./gradlew clean buildRelease step.
  5. Wait for the Azure Pipelines CI build to succeed. (This takes about half an hour.) If it fails, commit fixes, then wait again, and start over at the ./gradlew clean buildRelease step.
  6. Tag the release. 
      git tag -a v$NEWVERSION -m "Randoop version $NEWVERSION"
  git push --tags
  7. Follow the GitHub instructions for creating a release; in particular, start at https://github.com/randoop/randoop/releases, click "Draft a new release", in the "Tag" text field, choose v4.3.2, name it "Randoop version 4.3.2", use the changelog entry as the description (check the preview of the Markdown formatting, e.g., remove line breaks), upload the zip file from build/distributions, upload the jar files from build/libs, and, finally, click "publish release".
  8. Email an announcement to randoop-discuss@googlegroups.com
    Subject: Randoop version 4.3.2
    Randoop 4.3.2 has been released and is available at https://github.com/randoop/randoop/releases/latest. The changelog appears below.
  9. Make pull requests against downstream projects that use Randoop. In each, search for the old version number.

Publishing artifacts to Maven repositories

To publish jars to Maven repositories:

Randoop internals

This section describes Randoop's main concepts, data structures and internal algorithms.

Unit test concepts

A unit test is a snippet of code that checks some expected property of some classes under test. It is helpful to think of a unit test as consisting of two things:

As a simple example, suppose we are testing the JDK's collections classes (LinkedList, Collections, etc.) The following unit test checks that the size of a linked list is computed correctly after an element is added. 

1.   // Tests that the size of a list is 1 after adding an element.
2.   public void testAdd() {
3.      LinkedList<String> list = new LinkedList<>();
4.      boolean b = list.add(null);
5.      assertEquals(1, list.size());
6.   }

The test input consists of lines 3-4, which exercises the code of the list class by creating a list, and then calling the add method. The test check is on line 5.

In Randoop, a test input is represented as a Sequence, and test checks are represented using Checks. Each sequence is made up of one or more Statement objects.

Randoop's generation algorithms creates Sequences, adding the appropriate Checks to each, and output the results to the user as unit tests.

Sequences

Randoop generates Sequence objects, which are used to construct the inputs of unit tests by adding checks. In Randoop, all test inputs are sequences of operations that use the methods and constructors of the classes under test.

Continuing with our running example, here is an example of a sequence of statements that calls methods from the classes under test: separate line: 

  LinkedList<String> l = new LinkedList<>();
  String str = "hi!"
  l.addFirst(str);
  int i = l.size();
  TreeSet<String> t = new TreeSet<>(l);
  Set<String> s = Collections.synchronizedSet(t);

Notice that each statement has three elements:

  1. A specific method (or constructor) that is being called.
  2. A value (primitive or object) returned by each call (l, str, i, t, and s).
  3. Inputs to the call, all of which come from values produced in earlier statements.

Rearranging the above sequence, we can see these elements more clearly: 

                  Result      Operation        Inputs
                  ======      =============        ======
statement 0:      l       =   new LinkedList       ()
statement 1:      str     =   "hi!"                ()
statement 2:                  addFirst             (l, str)
statement 3:      i       =   size                 (l)
statement 4:      t       =   new TreeSet          (l)
statement 5:      s       =   synchronizedSet      (t)
The three elements of a statement map to the following classes in Randoop:

Creating sequences

You can create sequences in a variety of ways. The first way is via extension: take a sequence and add a new statement at the bottom. Recall the example sequence: 

  LinkedList<String> l = new LinkedList<>();
  String str = "hi!";
  l.addFirst(str);
  int i = l.size();
  TreeSet<String> t = new TreeSet<>(l);
  Set s = Collections.synchronizedSet(t);

To create this sequence by extension we need the operations. Because this code involves parameterized types, we need to instantiate the generic class types that are used in order to get the type substitution that we need. For instance, for the LinkedList<String> constructor call, we write: 

  InstantiatedType linkedListType = JDKTypes.LINKED_LIST_TYPE.instantiate(ConcreteTypes.STRING_TYPE);
  Substitution<ReferenceType> substLL = linkedListType.getTypeSubstitution();
  TypedOperation newLL = TypedOperation.forConstructor(LinkedList.class.getConstructor()).substitute(substLL);

which first creates the parameterized type, gets the type substitution, and then applies the substitution to the operation created from the constructor of the generic class. The next operations aren't from generic classes, and don't require this amount of setup: 

  TypedOperation newOb = TypedOperation.createPrimitiveInitialization(ConcreteTypes.STRING_TYPE, "hi!");
  TypedOperation addFirst = TypedOperation.forMethod(LinkedList.class.getMethod("addFirst", Object.class)).substitute(substLL);
  TypedOperation size = TypedOperation.forMethod(LinkedList.class.getMethod("size")).substitute(substLL);

But operations with wildcards require a substitution be applied first for the variables in the type, a capture conversion be applied, and then a substitution for the variables from the capture conversion. 

  InstantiatedType treeSetType = JDKTypes.TREE_SET_TYPE.instantiate(ConcreteTypes.STRING_TYPE);
  Substitution<ReferenceType> substTS = treeSetType.getTypeSubstitution();
  TypedOperation wcTS = TypedOperation.forConstructor(TreeSet.class.getConstructor(Collection.class)).substitute(substTS).applyCaptureConversion();
  Substitution<ReferenceType> substWC = new Substitution(wcTS.getTypeParameters(), (ReferenceType)ConcreteTypes.STRING_TYPE);
  TypedOperation newTS = wcTS.substitute(substWC);

And, similarly a generic operation requires building a substitution for the parameter to the method: 

   TypedOperation syncA = TypedOperation.forMethod(Collections.class.getMethod("synchronizedSet", Set.class));
   Substitution<ReferenceType> substA = new Substitution(syncA.getTypeParameters(), (ReferenceType)ConcreteTypes.STRING_TYPE);
   TypedOperation syncS = syncA.substitute(substA);

Once the operations are defined with the proper types, we can build the sequence by extension: 

  Sequence s = new Sequence();
     s = s.extend(newLL);
     s = s.extend(newOb);
     s = s.extend(addFirst, s.getVariable(0), s.getVariable(1));
     s = s.extend(size,    s.getVariable(0));
     s = s.extend(newTS,   s.getVariable(0));
     s = s.extend(syncS,   s.getVariable(4));

A couple notable points:

A second way to create a sequence is using concatenation: given sequences s1, s2, and s3, you can create a new sequence that is the concatenation of the three sequences. 

List<Sequence> seqs = new ArrayList<Sequence>();
seqs.add(s1);
seqs.add(s2);
seqs.add(s3);
Sequence newSeq = Sequence.concatenate(seqs);

If you're wondering why concatenation is useful, it is actually how Randoop combines sequences to create new ones. To create a new sequence that tests method m(A a, B b), Randoop first finds and concatenates previously-generated sequences that create objects of type A and B, and then extends the concatenated sequence with a call of m. See Section 2.2. of this paper for more details.

A third way to create a sequence is by parsing it from a String. For example, given a string parseable with the following contents: 

var0 =  cons : java.util.LinkedList.<init>() :
var1 =  cons : java.lang.Object.<init>() :
var2 =  method : java.util.LinkedList.addFirst(java.lang.Object) : var0 var1
var3 =  method : java.util.LinkedList.size() : var0
var4 =  cons : java.util.TreeSet.<init>(java.util.Collection) : var0
var5 =  method : java.util.Collections.synchronizedSet(java.util.Set) : var4

The following call creates a sequence corresponding to our running-example sequence: 

Sequence seq = Sequence.parse(parseable);

Executable sequences

An ExecutableSequence wraps a Sequence and adds two pieces of functionality:

Executing a sequence

Suppose we have created a Sequence s The following two lines will execute the sequence: 

ExecutableSequence es = new ExecutableSequence(s);
es.execute(null);

After the execute method returns, you can access the runtime objects created during execution via the getResult(int i) method, which returns the result of executing statement i. For example, assuming s refers to our running-example sequence from above, the following code prints out 1, i.e. the return value of the call to method LinkedList.size(). 

  ExecutableSequence es = new ExecutableSequence(s);
  es.execute(null);

  // Assuming statement at index 3 returned normally, print the runtime value
  ExecutionOutcome resultAt3 = es.getResult(3);
  if (resultAt3 instanceof NormalExecution) {
    System.out.println(((NormalExecution)resultAt3).getRuntimeValue());
  }

To dig more into execution results, explore the classes Execution and ExecutionOutcome.

Miscellaneous notes

Writing/reading sequences to file

Writing a sequence as a JUnit test

Writing a list of executed sequences as JUnit files: 

  List<ExecutableSequence> sequences = ...;
  JunitFileWriter jfw =
  new JunitFileWriter(output_dir, junit_package_name, junit_classname, testsperfile);
  jfw.createJunitFiles(seq);

If you want to modify Randoop's JUnit-generating code, here are places to look:

Checks

A Check is an object that represents an expected property of a sequence. In terms of unit test concepts, a check represents some piece of checking code (or “test oracle”) of a unit test.

For example, consider the following unit test: 

 1.  // Tests that the size of a list is 1 after adding an element.
 2.  @Test
 3.  public void testAdd() {
 4.     LinkedList l = new LinkedList();            // Create a list.
 5.     assertEquals(l, l);                         // List should be equal to itself.
 6.     Object o = new Object();
 7.     boolean b = l.add(o);                       // Add an element to it.
 8.     org.junit.Assert.assertEquals(1, l.size()); // List should have size 1.
 9.     org.junit.Assert.assertEquals(l, l);        // List should still be equal to itself.
10.     int i = 10;
11.     try {
12.       Object o2 = l.remove(i);                  // Removing from invalid index
13.       org.junit.Assert.fail();                  // should throw exception.
14.     } catch (IndexOutOfBoundsException e) {
15.       // expected exception.
16.     }
17.  }

Like a typical unit test, this test combines test input code (the code creating a list, adding an element to it, removing an element, etc.) with checking code (the code in lines 6-7 creating assertions, and the code in lines 9, 11-14 checking that the correct exception is thrown). In Randoop, test input code is represented as a Sequence and checking code is represented by Checks associated with the sequence.

More specifically, each check is associated with an index in the sequence, where the index represents the time at which the check should be performed. For example, rearranging the above test to better see its structure, we have: 


                          Checks performed
Statement                 after statement
=========                 ===============
l  = new LinkedList()
                           l.equals(l) returns true.
o  = new Object()
                           no checks.
b  = l.add(o)
                           l.size() returns 1.
                           l.equals(l) returns true.
i = 10;
                           no checks.
o2 = l.remove(i)
                           throws IndexOutOfBoundsException.

The example illustrates how Randoop represents a test input along with its correctness checks.

Statements vs. checks

Looking at the above example, you may reasonably ask: why do we draw a distinction between “test input code” (the statements on the left) and “checking code” (the statements on the right)? After all, aren't the calls to size and equals in the checking column also calls of the classes under test, and why not consider them part of the test input?

We offer an answer in terms of how Randoop works. Randoop generates test inputs (Sequences) randomly, by combining and extending previously-generated sequences. On the other hand, Randoop performs its checks deterministically. For every sequence it creates, it performs checks on all the objects of the sequence.

Given how Randoop treats sequences and checks differently, it makes more sense for a call like l.equals(l) to be expressed as a Check, so that it is always (not randomly) performed.

Executing Checks

(Under construction)

Main entry points

Test generator classes