Randoop logo

Randoop Developer's Manual

(Also see the Randoop Manual.)


Getting started

Clone the repository:

  git clone git@github.com:randoop/randoop.git

When working on a particular feature, create a new branch and work there rather than in the master branch.

Prerequisites

Install the CSS::DOM Perl module.

Install html5validator, using one of the the following two command lines:

sudo pip install html5validator
pip install --user html5validator

Building Randoop

In Randoop's top-level directory, run

  ./gradlew build manual

This command compiles Randoop, compiles the Java agents, runs all tests, builds the jar files, and updates the manual from the source Javadoc. Generated files are placed in subdirectories named build in the parent directory and in each of the agent subdirectories. Jar files are placed in directory build/libs.

Windows users: you do not need to include the "./" when running Gradle. When our instructions say to run ./gradlew build, you will run gradlew build instead.

You can also break this process down by running separate tasks. For the full list of tasks supported by the Randoop build script, type ./gradlew tasks. Particular tasks that may be useful during development include

./gradlew jar to build the Randoop jar files
./gradlew assemble to compile and build Randoop and fat jar files
./gradlew singleJar to build the "user" jar including all library dependencies
./gradlew check to run all of the tests
./gradlew testReport to generate HTML reports from tests (in build/reports/allTests)
./gradlew clean to remove the build subdirectory
./gradlew compileJava to compile the main source set
./gradlew compileTestJava to compile the test source set
./gradlew compileAgentTestJava to compile the agentTest source set
./gradlew manual to update the manual from the source Javadoc

Compiling uses the concept of source sets. A source set is a grouping of source files that can have its own set of dependencies and tasks. For instance, in the Randoop build.gradle there is the agentTest source set that is separate from the test source set because the test classes must run using the exercised-class Java agent. The Gradle Java plugin normally has two source sets named main and test, and the Randoop build script defines others to better structure the build process.

Here are some Gradle tricks that may be useful:

The Gradle wrapper

Randoop is built with Gradle. The shell script named ./gradlew and the Windows batch script gradlew.bat run the Gradle wrapper for the project, which is the recommended way to run Gradle commands. Using the wrapper means you don't have to install Gradle yourself, instead the wrapper will download Gradle the first time you run a build command.

To learn more about Gradle beyond the basic commands given here, see the Gradle command-line tutorial.

The Randoop build script

The Gradle build tasks for Randoop are defined in the build script build.gradle, along with the build scripts for the java agents in the corresponding subproject directories. This is the file you would modify to change the Randoop build. Note that the script uses plugins that each define tasks that may not be explicit within the script, though it may be possible to configure them differently by following the instructions at the plugin sites.

If you add or change a build task in a way that affects how Randoop is built, you should also update this document, which appears in the gh-pages branch. When you push your changes to the master branch on GitHub, you should also push to the gh-pages branch.

Running Randoop

Randoop classpath

When running Randoop, you should use the file randoop-all-X.Y.Z.jar that includes all dependencies. For more detail, see the Randoop Manual.

Otherwise, to run Randoop, your classpath must include either the Randoop class files or the Randoop jar file, and also must include lib/plume.jar. After the build, the classpath should include either of the following

It is recommended that the classpath for classes under test be placed before Plume-lib, should there be a conflict in included classes.

The jar files for Java agents are located in the build/libs subdirectory of both the Randoop project and the agent project directories. So, for instance, the following will work to run the exercised-class agent:

-javaagent:<randoop-path>/build/libs/exercised-class-3.1.5.jar

Testing Randoop

The Travis tests currently pass if this icon is green: Randoop Travis status icon

This command will reveal most problems:

./gradlew clean assemble check manual

Travis also runs these tests from a fresh clone, which may reveal some problems that would not otherwise be exposed. Therefore, you can create a branch and push it to the Randoop repository. If it succeeds, you can merge it. If it fails, you will be notified by email, and you can fix it without having destabilized the master branch.

Adding tests

As you add new functionality to Randoop, you should also add new tests. Randoop has three kinds of tests: unit tests of Randoop internal classes (under directory src/test), unit tests of Randoop Java agents (under directory src/agenttest), and system tests that run Randoop on input classes.

Unit tests

If you write JUnit tests that change the static state of either a command-line argument or a static class, then you should add BeforeClass and AfterClass annotated methods that save and restore the prior state. For command-line arguments, this can be done by adding the following code to the beginning of the test class:


  private static OptionsCache optionsCache;

  @BeforeClass
  public static void setup() {
    optionsCache = new OptionsCache();
    optionsCache.saveState();
  }

  @AfterClass
  public static void restore() {
    optionsCache.restoreState();
  }

This uses the randoop.main.OptionsCache class to save the static state of all classes that include the definition of command-line arguments. For classes under test with static state that is changed, you would use the randoop.reflection.StaticCache class to save and restore the state of that particular class in a similar way.

System tests

The system tests are test methods of class randoop.main.RandoopSystemTest in the src/systemtest directory. Each test uses one or more input classes from the testInput source set, which resides in the src/testinput/java subdirectory. (System tests are run with a different classpath than the other tests, and will only take input classes from src/testinput.) So, to set up a new system test, you need to add any new input classes, and add a new test method to the system test class.

Here is an example test from randoop.main.RandoopSystemTest that illustrates the main themes that occur in writing a system test.


 @Test
 public void runLiteralsTest() {

   // 1. Set up a working directory -- choose a unique name
  TestEnvironment testEnvironment =
        systemTestEnvironment.createTestEnvironment("literals-test");

   // 2. Set options for test generation
  RandoopOptions options = RandoopOptions.createOptions(testEnvironment);
  options.setPackageName("");
  options.setRegressionBasename("LiteralsReg");
  options.setErrorBasename("LiteralsErr");
  options.setOption("inputlimit", "1000");
  options.addTestClass("randoop.literals.A");
  options.addTestClass("randoop.literals.A2");
  options.addTestClass("randoop.literals.B");
  options.setOption("literals-level", "CLASS");
  options.setOption("literals-file", "resources/systemTest/literalsfile.txt");

  // 3. Indicate whether you expect regression tests (if you don't care use ExpectedTests.DONT_CARE)
  ExpectedTests expectedRegressionTests = ExpectedTests.SOME;
  ExpectedTests expectedErrorTests = ExpectedTests.NONE;

  // 4. Run Randoop, compile and run generated tests, and check to see if expectations met
  generateAndTest(testEnvironment, options, expectedRegressionTests, expectedErrorTests);
 }

The options list is exactly what you would give at the command line, except for setting the names and package for generated tests. All Randoop options may be given, except for --junit-reflection-allowed=false since the test run method only looks for JUnit suite classes.

Most existing tests take this form, using the RandoopSystemTest.generateAndTest() method, which checks whether Randoop generates the expected tests, that the tests compile, and that regression tests pass and error tests fail when run. The method also allows indicating for both kinds of tests, whether there should be some (at least one) test, none, or if it doesn't matter. This is done using the ExpectedTests enumerated type, which has values SOME, NONE, and DONT_CARE.

However, it is possible to build a test differently using the utility methods of the RandoopSystemTest class. At a minimum, a system test always attempts to compile generated tests. It should also try to run any generated tests and confirm that regression tests pass, and that error tests fail. Additional information can be found in the RandoopSystemTest class.

Editing Randoop

Using an IDE

The build script creates files for use by IntelliJ IDEA, NetBeans, and Eclipse.

Other than .gitignore, the tool-specific settings are not part of the repository. If your tool creates new configuration files, please add them to the .gitignore file. If you edit build.gradle, please verify that it doesn't change behavior from the command line using the Gradle wrapper.

Documenting changes

Any user-visible changes should be documented in the changelog devnotes/CHANGES.txt. This text is used for documenting releases, so keeping it up-to-date and ready for public release saves time in the long run. Order the information by importance to the user. Be sure to describe the change in terms of how it affects the user. List all fixed bugs, which you can obtain by querying GitHub. There is no need to describe each one individually unless users need to know about them individually. Optionally, mention any serious reported issues that are not fixed.

Formatting Randoop source code

The Randoop source code is formatted in Google Java Format by using the google-java-format tool. The RandoopGradle build script is setup to allow this tool to be applied using the following commands: (Please read cautions below before running the tool.)

CAUTION: The first command will change every single Java file in the repository, and should not be done before submitting a Pull Request against a branch that is out of format. Instead, make sure that the base branch is reformatted before creating the PR.

Updating libraries

Periodically, follow the instructions in lib/README to update local libraries. (Background: Gradle downloads the appropriate version of most of the libraries that Randoop uses. However, Randoop also uses other local libraries that are kept in the lib directory.)

Modifying the manual

The documentation for Randoop is in the subdirectory src/docs/manual:

The command ./gradlew manual updates command-line argument documentation in the user manual, and updates the table-of-contents in both manuals (in the src/docs/manual directory).

The Javadoc is generated by the command ./gradlew javadoc, which places the files into build/docs/api/. These files are published to the project website by the publishSite task described below.

Additional documentation that is not published to the project website is located in devnotes. This includes the change log, diagrams, and project ideas documents.

Modifying the website

The website, including the manuals, can be found in src/docs. These files, along with the Javadoc API, are moved to the docs directory by the publishSite task.

Releasing a new version of Randoop

To make a release from the master branch, follow these steps to ensure there are no changes needed:

  1. Be sure that libraries are up-to-date.
  2. Commit any changes; push any unpushed commits. Find out if any exist by running:
    git status && git log --branches --not --remotes
  3. Compile and test. If any tests fail, fix them and start over.
    ./gradlew clean build
  4. Ensure that the Javadoc compiles correctly:
    ./gradlew javadoc validateAPI
  5. Ensure that the manual HTML is valid:
    ./gradlew manual validateManual

Once there are no more changes to make:

  1. Increment the Randoop version number:
    1. Set environment variables for the old and new version numbers. (Don't use the examples given here!)
      OLDVERSION=3.1.5
      NEWVERSION=3.1.6
      These are used for working with git.
    2. Update the version number and date in each of:
      • build.gradle
      • src/main/java/randoop/Globals.java
      • devnotes/CHANGES.txt
      • src/docs/manual/index.html (many locations)
      • src/docs/manual/dev.html (this document)
      TODO: automate this step
  2. Double-check that the changelog devnotes/CHANGES.txt is up-to-date by running a command
    git log v$OLDVERSION..
    and ensuring that no user-visible changes have been missed. Have a colleague review the changelog entry.
  3. Create the distribution files with ./gradlew clean buildRelease. This will run any tasks that are needed for the release, and will also replace the project site in the docs directory.
  4. Commit and push those changes.
      git commit . -m "Update version to $NEWVERSION"
      git push
    
  5. Make a repository tag:
      git tag -a v$NEWVERSION -m "Randoop version $NEWVERSION"
      git push --tags
  6. Follow the GitHub instructions for creating a release; in particular, start at https://github.com/randoop/randoop/releases, click "Tags", click "add release notes" for the new tag (e.g., v3.1.6), name it "Randoop version 3.1.6", use the changelog entry as the description (but remove any line breaks and check the preview), upload the jar files from build/libs, upload the zip file from build/distributions and, finally, click "publish release".
  7. Check that there are no broken links on the published website:
    ./gradlew checklink
    Any output is written to a file
    checklink-log.txt
    If there are any errors, verify that each link on the site is actually broken, and if so, fix the link, and rerun
    ./gradlew buildRelease
    and commit and push those changes. If the check succeeds, run:
    rm -f checklink-log.txt
  8. Email an announcement to randoop-discuss@googlegroups.com.
    Subject: Randoop version 3.1.6
    Randoop 3.1.6 has been released and is available at https://github.com/randoop/randoop/releases/latest. The changelog appears below.

Randoop internals

This section describes Randoop's main concepts, data structures and internal algorithms. To this end, it helps to understand our model of unit tests.

Unit test concepts

A unit test is a snippet of code that checks some expected property of some classes under test. It is helpful to think of a unit test as consisting of two things:

As a simple example, suppose we are testing the JDK's collections classes (LinkedList, Collections, etc.) The following unit test checks that the size of a linked list is computed correctly after an element is added.

1.   // Tests that the size of a list is 1 after adding an element.
2.   public void testAdd() {
3.      LinkedList<String> list = new LinkedList<>();
4.      boolean b = list.add(null);
5.      assertTrue(list.size() == 1);
6.   }

The test input consists of lines 3-4, which exercises the code of the list class by creating a list, and then calling the add method. The test check is on line 5.

In Randoop, a test input is represented as a Sequence, and test checks are represented using Checks. Each sequence is made up of one or more Statement objects.

Randoop's generation algorithms creates Sequences, adding the appropriate Checks to each, and output the results to the user as unit tests.

Sequences

Randoop generates Sequence objects, which are used to construct the inputs of unit tests by adding checks. In Randoop, all test inputs are sequences of operations that use the methods and constructors of the classes under test.

Continuing with our running example, here is an example of a sequence of statements that calls methods from the classes under test: separate line:

  LinkedList<String> l = new LinkedList<>();
  String str = "hi!"
  l.addFirst(str);
  int i = l.size();
  TreeSet<String> t = new TreeSet<>(l);
  Set<String> s = Collections.synchronizedSet(t);

Notice that each statement has three elements:

  1. A specific method (or constructor) that is being called.
  2. A value (primitive or object) returned by each call (l, str, i, t, and s).
  3. Inputs to the call, all of which come from values produced in earlier statements.

Rearranging the above sequence, we can see these elements more clearly:

                  Result      Operation        Inputs
                  ======      =============        ======
statement 0:      l       =   new LinkedList       ()
statement 1:      str     =   "hi!"                ()
statement 2:                  addFirst             (l, str)
statement 3:      i       =   size                 (l)
statement 4:      t       =   new TreeSet          (l)
statement 5:      s       =   synchronizedSet      (t)
The three elements of a statement map to the following classes in Randoop:

Creating sequences

You can create sequences in a variety of ways. The first way is via extension: take a sequence and add a new statement at the bottom. Recall the example sequence:

  LinkedList<String> l = new LinkedList<>();
  String str = "hi!";
  l.addFirst(str);
  int i = l.size();
  TreeSet<String> t = new TreeSet<>(l);
  Set s = Collections.synchronizedSet(t);

To create this sequence by extension we need the operations. Because this code involves parameterized types, we need to instantiate the generic class types that are used in order to get the type substitution that we need. For instance, for the LinkedList<String> constructor call, we write:

  InstantiatedType linkedListType = JDKTypes.LINKED_LIST_TYPE.instantiate(ConcreteTypes.STRING_TYPE);
  Substitution<ReferenceType> substLL = linkedListType.getTypeSubstitution();
  TypedOperation newLL = TypedOperation.forConstructor(LinkedList.class.getConstructor()).apply(substLL);

which first creates the parameterized type, gets the type substitution, and then applies the substitution to the operation created from the constructor of the generic class. The next operations aren't from generic classes, and don't require this amount of setup:

  TypedOperation newOb = TypedOperation.createPrimitiveInitialization(ConcreteTypes.STRING_TYPE, "hi!");
  TypedOperation addFirst = TypedOperation.forMethod(LinkedList.class.getMethod("addFirst", Object.class)).apply(substLL);
  TypedOperation size = TypedOperation.forMethod(LinkedList.class.getMethod("size")).apply(substLL);

But operations with wildcards require a substitution be applied first for the variables in the type, a capture conversion be applied, and then a substitution for the variables from the capture conversion.

  InstantiatedType treeSetType = JDKTypes.TREE_SET_TYPE.instantiate(ConcreteTypes.STRING_TYPE);
  Substitution<ReferenceType> substTS = treeSetType.getTypeSubstitution();
  TypedOperation wcTS = TypedOperation.forConstructor(TreeSet.class.getConstructor(Collection.class)).apply(substTS).applyCaptureConversion();
  Substitution<ReferenceType> substWC = Substitution.forArgs(wcTS.getTypeParameters(), (ReferenceType)ConcreteTypes.STRING_TYPE);
  TypedOperation newTS = wcTS.apply(substWC);

And, similarly a generic operation requires building a substitution for the parameter to the method:

   TypedOperation syncA = TypedOperation.forMethod(Collections.class.getMethod("synchronizedSet", Set.class));
   Substitution<ReferenceType> substA = Substitution.forArgs(syncA.getTypeParameters(), (ReferenceType)ConcreteTypes.STRING_TYPE);
   TypedOperation syncS = syncA.apply(substA);

Once the operations are defined with the proper types, we can build the sequence by extension:

  Sequence s = new Sequence();
     s = s.extend(newLL);
     s = s.extend(newOb);
     s = s.extend(addFirst, s.getVariable(0), s.getVariable(1));
     s = s.extend(size,    s.getVariable(0));
     s = s.extend(newTS,   s.getVariable(0));
     s = s.extend(syncS,   s.getVariable(4));

A couple notable points:

A second way to create a sequence is using concatenation: given sequences s1, s2, and s3, you can create a new sequence that is the concatenation of the three sequences.

List<Sequence> seqs = new ArrayList<Sequence>();
seqs.add(s1);
seqs.add(s2);
seqs.add(s3);
Sequence newSeq = Sequence.concatenate(seqs);

If you're wondering why concatenation is useful, it is actually how Randoop combines sequences to create new ones. To create a new sequence that tests method m(A a, B b), Randoop first finds and concatenates previously-generated sequences that create objects of type A and B, and then extends the concatenated sequence with a call of m. See Section 2.2. of this paper for more details.

A third way to create a sequence is by parsing it from a String. For example, given a string parseable with the following contents:

var0 =  cons : java.util.LinkedList.<init>() :
var1 =  cons : java.lang.Object.<init>() :
var2 =  method : java.util.LinkedList.addFirst(java.lang.Object) : var0 var1
var3 =  method : java.util.LinkedList.size() : var0
var4 =  cons : java.util.TreeSet.<init>(java.util.Collection) : var0
var5 =  method : java.util.Collections.synchronizedSet(java.util.Set) : var4

The following call creates a sequence corresponding to our running-example sequence:

Sequence seq = Sequence.parse(parseable);

Executable sequences

An ExecutableSequence wraps a Sequence and adds two pieces of functionality:

Executing a sequence

Suppose we have created a Sequence s The following two lines will execute the sequence:

ExecutableSequence es = new ExecutableSequence(s);
es.execute(null);

After the execute method returns, you can access the runtime objects created during execution via the getResult(int i) method, which returns the result of executing statement i. For example, assuming s refers to our running-example sequence from above, the following code prints out 1, i.e. the return value of the call to method LinkedList.size().

  ExecutableSequence es = new ExecutableSequence(s);
  es.execute(null);

  // Assuming statement at index 3 returned normally, print the runtime value
  ExecutionOutcome resultAt3 = es.getResult(3);
  if (resultAt3 instanceof NormalExecution) {
    System.out.println(((NormalExecution)resultAt3).getRuntimeValue());
  }

To dig more into execution results, explore the classes Execution and ExecutionOutcome.

Miscellaneous notes

Writing/reading sequences to file

Writing a sequence as a JUnit test

Writing a list of executed sequences as JUnit files:

  List<ExecutableSequence> sequences = ...;
  JunitFileWriter jfw =
  new JunitFileWriter(output_dir, junit_package_name, junit_classname, testsperfile);
  jfw.createJunitFiles(seq);

If you want to modify Randoop's JUnit-generating code, here are places to look:

Checks

A Check is an object that represents an expected property of a sequence. In terms of unit test concepts, a check represents some piece of checking code (or “test oracle”) of a unit test.

For example, consider the following unit test:

 1.  // Tests that the size of a list is 1 after adding an element.
 2.  @Test
 3.  public void testAdd() {
 4.     LinkedList l = new LinkedList();            // Create a list.
 5.     assertTrue(l.equals(l));                    // List should be equal to itself.
 6.     Object o = new Object();
 7.     boolean b = l.add(o);                       // Add an element to it.
 8.     org.junit.Assert.assertTrue(l.size() == 1); // List should have size 1.
 9.     org.junit.Assert.assertTrue(l.equals(l));   // List should still be equal to itself.
10.     int i = 10;
11.     try {
12.       Object o2 = l.remove(i);                  // Removing from invalid index
13.       org.junit.Assert.fail();                  // should throw exception.
14.     } catch (IndexOutOfBoundsException e) {
15.       // expected exception.
16.     }
17.  }

Like a typical unit test, this test combines test input code (the code creating a list, adding an element to it, removing an element, etc.) with checking code (the code in lines 6-7 creating assertions, and the code in lines 9, 11-14 checking that the correct exception is thrown). In Randoop, test input code is represented as a Sequence and checking code is represented by Checks associated with the sequence.

More specifically, each check is associated with an index in the sequence, where the index represents the time at which the check should be performed. For example, rearranging the above test to better see its structure, we have:


                          Checks performed
Statement                 after statement
=========                 ===============
l  = new LinkedList()
                           l.equals(l) returns true.
o  = new Object()
                           no checks.
b  = l.add(o)
                           l.size() returns 1.
                           l.equals(l) returns true.
i = 10;
                           no checks.
o2 = l.remove(i)
                           throws IndexOutOfBoundsException.

The example illustrates how Randoop represents a test input along with its correctness checks.

Statements vs. checks

Looking at the above example, you may reasonably ask: why do we draw a distinction between “test input code” (the statements on the left) and “checking code” (the statements on the right)? After all, aren't the calls to size and equals in the checking column also calls of the classes under test, and why not consider them part of the test input?

We offer an answer in terms of how Randoop works. Randoop generates test inputs (Sequences) randomly, by combining and extending previously-generated sequences. On the other hand, Randoop performs its checks deterministically. For every sequence it creates, it performs checks on all the objects of the sequence.

Given how Randoop treats sequences and checks differently, it makes more sense for a call like l.equals(l) to be expressed as a Check, so that it is always (not randomly) performed.

Executing Checks

(Under construction)

Main entry points

Test generator classes