Randoop logo

Randoop Manual

This is the manual for Randoop version 3.1.4, released April 11, 2017. The Randoop homepage is https://randoop.github.io/randoop/.

Contents:

There is a Maven plug-in with source at https://bitbucket.org/javydreamercsw/randoop-maven-plugin/.

Randoop.NET is a version of Randoop that works with Microsoft's .NET platform, such as the C# programming language.

There is a separate Randoop Developer's Manual.

Introduction

Writing tests is a difficult and time-consuming activity, and yet it is a crucial part of good software engineering. Randoop automatically generates unit tests for Java classes. Randoop has had many successful applications, especially with library classes (such as java.util). Randoop is used at companies like ABB and Microsoft, and on open-source projects.

Here is a JUnit test case generated by Randoop that reveals an error in OpenJDK (comments added by hand):


  // This test shows that the JDK collection classes
  // can create an object that is not equal to itself.
  @Test
  public static void test1() {

    LinkedList list = new LinkedList();
    Object o1 = new Object();
    list.addFirst(o1);

    // A TreeSet is an ordered collection. According to the API
    // documentation, this constructor call should throw a
    // ClassCastException because the list element is not Comparable. But
    // the constructor silently (and problematically) accepts the list.
    TreeSet t1 = new TreeSet(list);

    Set s1 = Collections.synchronizedSet(t1);

    // At this point, we have successfully created a set (s1)
    // that violations reflexivity of equality: it is not equal
    // to itself! This assertion fails at run time on OpenJDK.
    org.junit.Assert.assertTrue(s1.equals(s1));
  }

Randoop outputs two kinds of tests:

You should fix the errors revealed by the error-revealing tests suite then rerun Randoop, until it generates no more error-revealing tests. Then, you can run the regression tests whenever you change your source code, to be notified of any changes you make to the behavior of your program. You can always re-run Randoop to check for new errors, to produce tests for newly-written code, or to regenerate tests after a code change that causes desirable behavior-changes.

Randoop's tests are not designed to be short or easy to read — but you will only read them on the rare occasions when they fail and reveal a bug or a regression failure.

Installing Randoop

Download and unzip the file randoop-3.1.4.zip. Set an environment variable RANDOOP_PATH to the path of the unzipped archive, and set an environment variable RANDOOP_JAR to the location of randoop-all-3.1.4.jar within the unzipped archive.

Alternately, if you want to view or work with the source code, follow the instructions in the Getting Started section of the Randoop Developer's Manual.

You may wish to join the randoop-discuss@googlegroups.com mailing list so that you are notified of new releases.

Running Randoop

Run Randoop by invoking its main class randoop.main.Main:

java -ea randoop.main.Main command args...

The classpath needs to contain both Randoop (randoop-all-3.1.4.jar) and the classes under test. For instance, if the current working directory contains a directory named bin containing your compiled classes, then you would use:

  ... -classpath bin:$(RANDOOP_JAR) ...

(On Windows, adjust the classpath, such as using semicolon instead of colon as the separator.) Note that you don't need the -classpath argument if you add $(RANDOOP_JAR) to your classpath.

Randoop supports two commands:

Generating tests

By default, Randoop generates and then outputs two kinds of unit tests, written to separate files.

Other generated tests, classified as invalid, are discarded.

This section of the manual gives an example of how to run Randoop, then describes test classification, error-revealing tests, and regression tests, and finally lists Randoop's command-line options.

Example: Generating tests for java.util.Collections

Imagine we want to generate tests for the class java.util.Collections.

Randoop's tests only use classes you specify. In order to effectively test Collections, you should probably also specify some helper classes, including classes that generate collections. For this example, we will add java.util.TreeSet to the mix.

Create a file myclasses.txt that lists the names of the classes under test:

java.util.Collections
java.util.TreeSet

and invoke Randoop as follows:

java -classpath $(RANDOOP_JAR) randoop.main.Main gentests --classlist=myclasses.txt --timelimit=60

After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

Created file: my/home/directory/RegressionTest0.java
Created file: my/home/directory/RegressionTest.java
Created file: my/home/directory/ErrorTest0.java
Created file: my/home/directory/ErrorTest.java
done.

Randoop creates two different main test suites, in classes ErrorTest and RegressionTest. Compile and run the tests. (The classpath should include the code under test, the generated tests, and JUnit files junit.jar and hamcrest-core.jar.)

export JUNITPATH=.../junit.jar:.../hamcrest-core.jar
javac -classpath .:$JUNITPATH ErrorTest*.java RegressionTest*.java
java -classpath .:$JUNITPATH org.junit.runner.JUnitCore ErrorTest
java -classpath .:$JUNITPATH org.junit.runner.JUnitCore RegressionTest

All of the tests in ErrorTest will fail, and all of the tests in RegressionTest will pass.

Specifying methods and constructors that may appear in a test

Randoop only invokes a method or constructor M if all of the following are true:

  1. M is declared in a class that is named by --testclass or --classlist, or is a member named by --methodlist.
  2. M does not match the pattern given via --omitmethods.
  3. All of the following are accessible from the package of the tests (set with --junit-package-name): M's class, M, and M's return type and parameter types.

A method or constructor M that satisfies the above conditions is called a "method under test". Randoop only calls methods under test; that is, these are the only methods that will appear in a test case.

The methods under test must include not only the methods whose results you wish to appear in an assert statement, but also methods that a test may need to call in order to set up state or create argument values. All of those methods are also being implicitly tested by the assert statement.

Use raw type names such as List, not parameterized types such as List<String>, when specifying classes or methods.

Do not include methods that side-effect global state. (Methods that side-effect the state of their receiver or other arguments are fine.) The reason is that such methods create dependent tests that cannot be run in isolation or reordered. For example, if test A side-effects global state and test B reads global state, then whether test B fails depends on whether test A is run before test B.

WARNING: If you ask Randoop to test code that modifies your file system (such as File.delete()), then Randoop will generate tests that modify your file system! Be careful when choosing classes and methods to test.

Classifying tests

Randoop generates many tests internally, and it classifies each generated test as error-revealing, expected behavior, or invalid. The classification depends primarily on whether the last statement throws an exception or violates a contract.

You can use command-line arguments to customize Randoop's rules for classifying tests.

Additional ERROR behaviors are described in the error-revealing tests section.

The default behavior for --unchecked-exception is to treat throwing any unchecked exception as expected, normal behavior. An alternative would be to consider a test that throws AssertionError as error-revealing. However, a random test generator often supplies illegal arguments (such as null to a method that requires a non-null argument), and this approach will lead to false alarms in which Randoop outputs an "error-revealing test" that misuses the software under test. Depending on your needs, you can override Randoop's default.

A thrown exception is not considered a contract violation unless the exception is tagged as error behavior. This is because a method may have a precondition that is unknown to Randoop. For example, a binary search implementation might require that its input is sorted. Randoop should not mark the method as buggy just because it throws an exception when Randoop passes it an ill-formed unsorted array. Randoop has no way of knowing each method's precondition, so by default it assumes that any exception thrown by a method is correct behavior in response to the values that it was passed. You can customize how exceptions are considered by assigning the exception behavior type.

You can also provide specifications of your code to further help Randoop classify tests; see the checkRep mechanism.

Error-revealing tests

When Randoop includes a error-revealing test in file ErrorTest.java, the test shows that the code violates its specification or contract — some property that it is supposed to preserve. For example, the test shown in the Introduction creates a TreeSet that violates reflexivity of equality: namely, for every object o, o.equals(o) should return true. The test reveals an error in the TreeSet constructor, which does not properly check its arguments (and possibly a second error in the TreeSet.equals method, which should always check if its argument is the same object as itself and return true in such a case).

The test in the Introduction is nicely succinct: it shows a small number of method calls leading up to the assertion violation. Because Randoop's generation is random, the actual test revealing the error can be significantly longer, and contain many irrelevant calls that are not necessary to reveal the contract violation. You may wish to minimize the test case to ease debugging.

What kinds of errors does Randoop check for?

When Randoop calls a method that creates an object, Randoop verifies that the object is well-formed. Currently, Randoop checks for the following contracts:

Violation of any of these contracts is highly likely to indicate an error.

You can add additional contracts to Randoop.

Regression tests

The tests in the RegressionTest*.java files record the current behavior (values returned or exceptions thrown) of the classes under test. These tests assert that behavior, and they currently pass. After you change your code, you can run these regression tests, which will alert you if your code changes affect the external behavior of the classes.

Here is an example of a regression test for the TreeSet class:

// This test passes when executed
@Test
public void test10() throws Throwable {

  java.util.TreeSet var0 = new java.util.TreeSet();
  java.lang.Short var1 = new java.lang.Short((short)100);
  boolean var2 = var0.contains(var1);

  // Regression assertion (captures the current behavior of the code)
  org.junit.Assert.assertTrue(var2 == false);
}

This test would fail if you introduced an error that caused contains to return true on an empty set.

A regression test can also assert what exception is currently thrown, and it will fail if the code is changed to no longer throw that exception.

Regression test failures

If a regression test fails, there are three possible causes. You need to debug the test failure to determine which is the cause, and then take the appropriate action.

Usually, a change to your program causes many test failures, not just one. This is because Randoop generates many tests for each part of your program's functionality. Don't panic! In most cases, all of the failures have the same cause.

You may wish to verify the cause of the failures. A way to do this is to change the part of the program, that you suspect is the cause, back to its previous behavior. Once you are satisfied that you understand the failures, you can discard the tests and have Randoop generate new ones. For example, suppose that you believe that all the test failures are caused by a change to method m. You can change m back to its previous implementation and run the tests. If they all succeed, you can change m back to its new implementation and re-generate the tests. (As an alternative to making a temporary change to your source code, you can use the map_calls feature of the Randoop agent to replace the new functionality by the old functionality.)

Customizing Randoop's behavior to your application

If you run Randoop without any command-line arguments or customizations to your environment, it will produce helpful tests. However, Randoop will produce much more helpful tests if you spend a little bit of time to tell Randoop about your application, and that is how we recommend using Randoop.

Command-line options

[+] marked option can be specified multiple times

Avoiding calls to specific methods

The code under test may make calls to certain methods that you do not want Randoop to execute. Two examples are System.exit (which will terminate not just the target program but also Randoop!) and methods that require user interaction, such as popping up a modal dialog box.

You can prevent Randoop from calling certain methods in the code under test, by using the --omitmethods=regexp command-line option.

Alternately, you can provide a different implementation of a method that will be used when running under Randoop. Randoop transforms the code under test, replacing each call to a undesired method by a call to a new implementation that you provide.

To transform calls to undesired methods, use the Java agent in the mapcall-3.1.4.jar file (either in the unzipped distribution file, or downloaded from the latest release). Then use this file pass the following command-line option to java when running Randoop:

    -javaagent:$(RANDOOP_PATH)/mapcall-3.1.4.jar=--map_calls=map_calls.txt

(Note: If you supply the -javaagent option to java when running Randoop, and there is any chance that Randoop may spawn another process (for example, if the --compare-checks command-line option is specified), then then you should also specify the --agent argument to Randoop.)

You need to supply the map_calls.txt file, and also a definition of the replacement methods. When you run the generated tests, you will also need to supply the same -javaagent:... argument. (In the future, Randoop may provide a way to generate standalone tests even when generating tests using the -javaagent:... argument.)

The map_calls.txt file consists of sections, where each section has the format

class-regex
  old-method-name (arg-type, ...) new-class-name
  old-method-name (arg-type, ...) new-class-name
  ...

Each section gives a regular expression that matches class names, and one or more replacements that will be done in classes that match the regexp. The replacement is the fully-qualified method name, fully-qualified argument types, and a new class name. A call to the old method will be replaced by a call to a method with identical name and arguments in the new class.

Blank lines and // comments are ignored in the map_calls.txt file.

See an example map_calls.txt file and an example definition of replacement methods.

Determining which calls to place in the map_calls file

If Randoop is calling methods that it should not (for example, while generating tests, a dialog box pops up), then you need to add more calls to you map_calls file. While the dialog box is displayed, determine which calls are executing by examining the stack trace. On Unix this can be done by typing Ctrl-\ (hold down the Control key, then press the Backslash key).

To determine the specific method, examine the application's code where it calls the method that puts up the box (or other unwanted activity). If it is a virtual call, the map must specify the name of the type used in the call.

Specifying representation invariant methods (such as checkRep)

The "representation invariant" of a class is a property that all instances of the class must satisfy. For example, a field might contain a non-negative number, or two fields might contain lists of the same length. Sometimes, a program contains a method that checks the rep invariant, for debugging purposes. If the rep invariant is ever violated, the program has a bug.

By using the @CheckRep annotation (don't forget to import randoop.*), you can tell Randoop which methods in your classes under test are rep invariant methods. Randoop will call these methods; if the method ever fails, Randoop outputs the test as a failing test.

A method annotated with @CheckRep must have one of two allowed signatures:

Instrumenting classes for filtering tests on exercised-classes

Randoop will ordinarily generate tests involving all accessible constructors, fields and methods of tests given using the --testclass and --classlist options. These can be restricted to tests that exercise the constructors and methods of classes from a different list given by using the --include-if-class-exercised option. However, for this option to work, the classes must be instrumented to indicate when they are exercised. To instrument classes for this filter, also use the exercised-class-3.1.4.jar by giving the following command-line option for java


  -javaagent:$(RANDOOP_PATH)/exercised-class-3.1.4.jar

This will instrument any non-abstract constructor and method of loaded classes so that the use of a class can be detected. (The instrumentation is not done on any JDK or JUnit classes.) (The exercised-class-3.1.4.jar file can be found in the unzipped distribution archive, or can be downloaded from the latest release.)

Specifying additional primitive values

By default, Randoop uses the following pool of primitive values as inputs to methods:

This is the initial set of seed values used by Randoop: in addition to the above values, any primitive value returned at runtime from a method call under test may be later used as input to another method call.

There are two ways of specifying additional seed values to Randoop.

Speeding up test generation

If you're interested in increasing the number of tests that Randoop generates in a given amount of time, you can try running Randoop with the option --usethreads=false. This will cause Randoop to use a single thread to execute all tests, and can speed up the tool by an order of magnitude.

By default, Randoop executes generated tests in separate threads, so that it can kill non-terminating threads that might otherwise prevent Randoop from making progress. For this reason, if you set --usethreads to false and Randoop appears to hang, it may be due to a randomly-generated test that is taking too long to execute and cannot be killed by Randoop.

Getting help

You can get help on Randoop in the following ways:

Troubleshooting

Randoop does not run

If the command to run Randoop has the output

Cannot find or load randoop.main.Main

then Randoop is not properly included in the classpath. One possibility is that you have included the Randoop jar file in the path specified by the CLASSPATH variable, but have given the location of the classes-under-test with the -classpath option at the command line. In this case, the java command will only use the classpath from -classpath.

It is recommended that both Randoop and the classes under test be included as the argument to the -classpath option to the java command. It only makes sense to set CLASSPATH to run Randoop if the location of classes-under-test will be the same each time you run it. See the Java Tutorial on path and classpaths.

If the command to run Randoop has the output

Cannot find the Java compiler. Check that classpath includes tools.jar

then you have installed Java incorrectly. You perhaps have the JRE on your path before the JDK. Ensure that JAVA_HOME appears at the front of your path. If you run the Windows operating system, see the instructions for setting a custom Java path after installing JDK 8.

Randoop cannot find class-under-test

If when Randoop is run, something like the following is output

No class found for type name "example.ExampleClass"

then the classes under test are not in the classpath. The most likely case is that you have not included the path to these classes. Otherwise, the classpath is not given properly. In particular, if you are running Randoop on Windows, parts of the classpath should be separated by a semi-colon ";" instead of the colon ":" shown in the examples in this document. See the Java Tutorial on paths and classpaths for platform specific information on paths.

Randoop does not create tests

If Randoop outputs

No tests were created. No JUnit class created.

then you need to change Randoop's parameters and re-run it. You might give Randoop more classes to analyze, a larger timeout, additional primitive values, specific methods to avoid calling, or some other configuration change. See elsewhere in this manual for suggestions.

Randoop does not terminate

If Randoop does not terminate, run it with the --log=filename command-line option, and either examine the log or send it to the Randoop developers for diagnosis.

You can also determine which calls are executing by examining the stack trace. On Unix this can be done by typing Ctrl-\ (hold down the Control key, then press the Backslash key).

Randoop produces different output on different runs

Randoop is deterministic: if you run it twice with the same arguments, it will produce the identical test suite. You can use the --randomseed command-line argument to make Randoop produce different tests.

If running Randoop twice produces different test suites, then there is nondeterminism in your program. Some ways to eliminate nondeterminism in a sequential Java program are:

Randoop stopped because of a flaky test

Randoop extends previously-generated tests by adding new method calls to them. If, in the new longer test, one of the embedded subtests throws an exception (even though when previously run on its own the subtest passed), then we say the test is flaky.

Flaky tests are usually due to global side effects, such as setting a static field; a Randoop-generated test might set the field directly or call a method that sets the field. Such methods are not appropriate for randomly-generated tests, and so you should not call Randoop on such methods. Calling Randoop on a method that side-effects local state, such as a field of the receiver or of an argument, is fine.

If Randoop encounters a flaky test, Randoop halts so that you can correct your invocation of Randoop, excluding the global-side-effecting method from consideration. You can do so by passing the --omitmethods command-line option.

To help you determine which method to omit, when logging is turned on (using the --log option) Randoop logs the list of method calls and field manipulations since the first execution of the subsequence where the exception occurred. You should determine which of these methods affected global state, and then exclude it by using the --omitmethods command-line option.

If you find a flaky test that does not result from a method that side-effects global state, then please help us improve Randoop by submitting an issue via the issue tracker to let us know what you encountered. (Randoop already discards any test with a flaky occurrence of an OutOfMemoryError or StackOverflowError as invalid. Additional information can help us find other heuristics that avoid generating flaky tests, or improve this documentation.)

If you are unable to determine which method caused the flaky tests and exclude it using the --omitmethods command-line option, then you can resort to the command-line argument --ignore-flaky-tests to discard the flaky tests. This is a last resort because it is better to avoid generating flaky tests rather than discarding them after generating them.

Tests behave differently in isolation or when reordered

Another symptom of calling Randoop on methods that have global side effects, in addition to flaky tests, is dependent tests: tests that behave differently depending on whether some other test has already run. For instance, test B might succeed if run after test A, but fail if run on its own.

You can either always run Randoop's generated tests in the same order (without performing test selection, prioritization, or parallelization), or you can avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

Another possible symptom is that if you run Randoop's tests and then your original test suite, the original test suite might fail because of side effects performed by Randoop's tests. To avoid this problem, you can run Randoop's tests in a separate JVM, or run Randoop's tests last, or avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

Credits

The following individuals have contributed code to Randoop: Ayman Abdelghany, Michael Ernst, Waylon Huang, Laura Inozemtseva, René Just, Peter Kalauskas, Ben Keller, Carlos Pacheco, Jeff Perkins, Sai Zhang.

The feedback of Randoop users has been very valuable.

If your name has been inadvertently omitted from this section, please let us know so we can correct the oversight.