Randoop logo

Randoop Manual

This is the manual for Randoop version 4.0.0, released February 28, 2018. The Randoop homepage is https://randoop.github.io/randoop/.


There is a Maven plug-in with source at https://bitbucket.org/javydreamercsw/randoop-maven-plugin/.

Randoop.NET is a version of Randoop that works with Microsoft's .NET platform, such as the C# programming language.

There is a separate Randoop Developer's Manual.


Writing tests is a difficult and time-consuming activity, and yet it is a crucial part of good software engineering. Randoop automatically generates unit tests for Java classes. Randoop has had many successful applications, especially with library classes (such as java.util). Randoop is used at companies like ABB and Microsoft, and on open-source projects.

Here is a JUnit test case generated by Randoop that reveals an error in OpenJDK (comments added by hand):

  // This test shows that the JDK collection classes
  // can create an object that is not equal to itself.
  public static void test1() {

    LinkedList list = new LinkedList();
    Object o1 = new Object();

    // A TreeSet is an ordered collection. According to the API
    // documentation, this constructor call should throw a
    // ClassCastException because the list element is not Comparable. But
    // the constructor silently (and problematically) accepts the list.
    TreeSet t1 = new TreeSet(list);

    Set s1 = Collections.synchronizedSet(t1);

    // At this point, we have successfully created a set (s1)
    // that violations reflexivity of equality: it is not equal
    // to itself! This assertion fails at run time on OpenJDK.

Randoop outputs two kinds of tests:

You should fix the errors revealed by the error-revealing tests suite then rerun Randoop, until it generates no more error-revealing tests. Then, you can run the regression tests whenever you change your source code, to be notified of any changes you make to the behavior of your program. You can always re-run Randoop to check for new errors, to produce tests for newly-written code, or to regenerate tests after a code change that causes desirable behavior-changes.

Randoop's tests are not designed to be short or easy to read — but you will only read them on the rare occasions when they fail and reveal a bug or a regression failure.

Installing Randoop

Download and unzip the file randoop-4.0.0.zip. Set an environment variable RANDOOP_PATH to the path of the unzipped archive, and set an environment variable RANDOOP_JAR to the location of randoop-all-4.0.0.jar within the unzipped archive.

Alternately, if you want to view or work with the source code, follow the instructions in the Getting Started section of the Randoop Developer's Manual.

You may wish to join the randoop-discuss@googlegroups.com mailing list so that you are notified of new releases.

Running Randoop

Run Randoop by invoking its main class randoop.main.Main:

java randoop.main.Main command args...

Randoop supports three commands:

(On Windows, adjust the classpath, such as using semicolon instead of colon as the separator.) You don't need the -classpath argument if you add $(RANDOOP_JAR) to your classpath.

Generating tests

By default, Randoop generates and then outputs two kinds of unit tests, written to separate files.

Other generated tests, classified as invalid, are discarded.

This section of the manual gives an example of how to run Randoop, then describes test classification, error-revealing tests, and regression tests, and finally lists Randoop's command-line options.

Example: Generating tests for java.util.Collections

Imagine we want to generate tests for the class java.util.Collections.

Randoop's tests only use classes you specify. In order to effectively test Collections, you should probably also specify some helper classes, including classes that generate collections. For this example, we will add java.util.TreeSet to the mix.

Create a file myclasses.txt that lists the names of the classes under test:


and invoke Randoop as follows:

java -classpath $(RANDOOP_JAR) randoop.main.Main gentests --classlist=myclasses.txt --time-limit=60

After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

Created file: my/home/directory/RegressionTest0.java
Created file: my/home/directory/RegressionTest.java
Created file: my/home/directory/ErrorTest0.java
Created file: my/home/directory/ErrorTest.java

Randoop creates two different main test suites, in classes ErrorTest and RegressionTest. Compile and run the tests. (The classpath should include the code under test, the generated tests, and JUnit files junit.jar and hamcrest-core.jar. Classes in java.util.* are always on the Java classpath, so they are not needed in this particular example, but are shown because you will usually need to supply them.)

export JUNITPATH=.../junit.jar:.../hamcrest-core.jar
javac -classpath .:$JUNITPATH ErrorTest*.java RegressionTest*.java -sourcepath .:path/to/files/under/test/
java -classpath .:$JUNITPATH:myclasspath org.junit.runner.JUnitCore ErrorTest
java -classpath .:$JUNITPATH:myclasspath org.junit.runner.JUnitCore RegressionTest

All of the tests in ErrorTest will fail, and all of the tests in RegressionTest will pass.

You also might want to minimize the error-revealing tests before running them:

java -cp .:junit.jar:myclasspath:.../randoop/build/libs/randoop-all-4.0.0.jar \
  randoop.main.Main minimize --suitepath=ErrorTest0.java --suiteclasspath=.:junit-4.11.jar:myclasspath

Specifying methods, constructors, and fields that may appear in a test

Randoop only invokes a method or constructor M if all of the following are true:

  1. M is declared in a class that is named by --testclass or --classlist, or is a member named by --methodlist.
  2. M does not match any of the patterns given via --omitmethods. or --omitmethods-list.
  3. All of the following are accessible from the package of the tests (set with --junit-package-name): M's class, M, and M's return type and parameter types.

A method or constructor M that satisfies the above conditions is called a "method under test". Randoop only calls methods under test; that is, these are the only methods that will appear in a test case.

The methods under test must include not only the methods whose results you wish to appear in an assert statement, but also methods that a test may need to call in order to set up state or create argument values. All of those methods are also being implicitly tested by the assert statement.

Use raw type names such as List, not parameterized types such as List<String>, when specifying classes or methods.

Do not include methods that side-effect global state. (Methods that side-effect the state of their receiver or other arguments are fine.) The reason is that such methods create dependent tests that cannot be run in isolation or reordered. For example, if test A side-effects global state and test B reads global state, then whether test B fails depends on whether test A is run before test B.

WARNING: If you ask Randoop to test code that modifies your file system (such as File.delete()), then Randoop will generate tests that modify your file system! Be careful when choosing classes and methods to test.

Classifying tests

Randoop generates many tests internally, and it classifies each generated test as error-revealing, expected behavior, or invalid. The classification depends primarily on whether the last statement throws an exception or violates a contract.

You can use command-line arguments to customize Randoop's rules for classifying tests.

Additional ERROR behaviors are described in the error-revealing tests section.

The default behavior for --unchecked-exception is to treat throwing any unchecked exception as expected, normal behavior. An alternative would be to consider a test that throws AssertionError as error-revealing. However, a random test generator often supplies illegal arguments (such as null to a method that requires a non-null argument), and this approach will lead to false alarms in which Randoop outputs an "error-revealing test" that misuses the software under test. Depending on your needs, you can override Randoop's default.

A thrown exception is not considered a contract violation unless the exception is tagged as error behavior. This is because a method may have a precondition that is unknown to Randoop. For example, a binary search implementation might require that its input is sorted. Randoop should not mark the method as buggy just because it throws an exception when Randoop passes it an ill-formed unsorted array. Randoop has no way of knowing each method's precondition, so by default it assumes that any exception thrown by a method is correct behavior in response to the values that it was passed. You can customize how exceptions are considered by assigning the exception behavior type.

You can also provide specifications of your code to further help Randoop classify tests; see the checkRep mechanism.

Error-revealing tests

When Randoop includes a error-revealing test in file ErrorTest.java, the test shows that the code violates its specification or contract — some property that it is supposed to preserve. For example, the test shown in the Introduction creates a TreeSet that violates reflexivity of equality: namely, for every object o, o.equals(o) should return true. The test reveals an error in the TreeSet constructor, which does not properly check its arguments (and possibly a second error in the TreeSet.equals method, which should always check if its argument is the same object as itself and return true in such a case).

The test in the Introduction is nicely succinct: it shows a small number of method calls leading up to the assertion violation. Because Randoop's generation is random, the actual test revealing the error can be significantly longer, and contain many irrelevant calls that are not necessary to reveal the contract violation. You may wish to minimize the test case to ease debugging.

What kinds of errors does Randoop check for?

When Randoop calls a method that creates an object, Randoop verifies that the object is well-formed. Currently, Randoop checks for the following contracts:

Violation of any of these contracts is highly likely to indicate an error.

You can add additional contracts to Randoop.

Regression tests

The tests in the RegressionTest*.java files record the current behavior (values returned or exceptions thrown) of the classes under test. These tests assert that behavior, and they currently pass. After you change your code, you can run these regression tests, which will alert you if your code changes affect the external behavior of the classes.

Here is an example of a regression test for the TreeSet class:

// This test passes when executed
public void test10() throws Throwable {

  java.util.TreeSet var0 = new java.util.TreeSet();
  java.lang.Short var1 = new java.lang.Short((short)100);
  boolean var2 = var0.contains(var1);

  // Regression assertion (captures the current behavior of the code)
  org.junit.Assert.assertTrue(var2 == false);

This test would fail if you introduced an error that caused contains to return true on an empty set.

A regression test can also assert what exception is currently thrown, and it will fail if the code is changed to no longer throw that exception.

Regression test failures

If a regression test fails even though you have not changed your program, then the test is flaky.

If a regression test fails after you change your program, there are three possible causes. You need to debug the test failure to determine which is the cause, and then take the appropriate action.

Usually, a change to your program causes many test failures, not just one. This is because Randoop generates many tests for each part of your program's functionality. Don't panic! In most cases, all of the failures have the same cause.

You may wish to verify the cause of the failures. A way to do this is to change the part of the program, that you suspect is the cause, back to its previous behavior. Once you are satisfied that you understand the failures, you can discard the tests and have Randoop generate new ones. For example, suppose that you believe that all the test failures are caused by a change to method m. You can change m back to its previous implementation and run the tests. If they all succeed, you can change m back to its new implementation and re-generate the tests. (As an alternative to making a temporary change to your source code, you can use the replacecall feature of the Randoop agent to replace the new functionality by the old functionality.)

Stages of test generation, and limiting test generation

Here is how Randoop attempts to generate a test:

  1. Randoop selects a method to call. (This is called a "step".) The selected method will be the last method call in the test. If Randoop is unable to find arguments to the method call (that are different from calls to the method in other tests), then Randoop starts over. Otherwise, Randoop has created a test candidate -- a new, unique sequence of operations that ends with the selected method.
  2. Randoop executes the test candidate and classifies its behavior. If its behavior is invalid, then Randoop discards it and starts over.
  3. Randoop outputs error-revealing and regression tests, in separate suites.

The user can provide limits on any part of the test generation process, or on Randoop's run time. Test generation stops when any of the following occurs (this is controlled by method AbstractGenerator.shouldStop):

Furthermore, see the command-line options for threading for additional per-test time limits.

Minimizing a failing JUnit test suite

Randoop includes a minimizer that, given a failing JUnit test suite file, produces a smaller one that fails in the same way as the original, having the same failing assertions with the same stack trace. Minimized test suites usually contain test cases that are easier to understand and debug. The minimizer can be used with any JUnit test suite: it can minimize failing tests that Randoop did not create.

The minimizer works by removing and simplifying methods and statements in its input file, without changing the test suite's failure behavior.

To minimize all failing tests in ErrorTest1.java, run Randoop as follows:

java -cp randoop-all-4.0.0.jar randoop.main.Main minimize \
  --suitepath=ErrorTest0.java --suiteclasspath=junit.jar:commons-lang.jar --testsuitetimeout=30

In the example above, the output file produced by the minimizer would be named ErrorTest1Minimized.java and the main class would be named ErrorTest1Minimized. In general, the minimizer will produce a Java file named InputTestSuiteMinimized.java where InputTestSuite.java is the name of the file being minimized. The main class within the Java file will be named InputTestSuiteMinimized.

It is desirable for the minimizer to retain both the original error-revealing test suite and the minimized test suite. This is because the minimizer may simplify a test suite in such a way that important information is lost. For example, when a test case fails due to a NullPointerException, the minimizer can replace the right hand side of an expression with the value null. This is sometimes an oversimplification.

See the documentation for the command-line options for failing test suite minimization.

Customizing Randoop's behavior to your application

If you run Randoop without any command-line arguments or customizations to your environment, it will produce helpful tests. However, Randoop will produce much more helpful tests if you spend a little bit of time to tell Randoop about your application, and that is how we recommend using Randoop.

Command-line options

[+] marked option can be specified multiple times

A Randoop "fully-qualified signature" is

where package-name is a period-separated list of identifiers, and argument-list is a comma-separated list of fully-qualified Java raw types, without spaces.

Avoiding calls to specific methods

The code under test can make calls to methods that you do not want executed. One is System.exit, which will terminate Randoop if called during test generation. Others, such as the JOptionPane.showConfirmDialog methods, open a modal dialog box requiring user interaction, which interferes with automated testing. It is not enough to use --omitmethods to prevent direct calls to these methods, because the methods may still be called indirectly. To prevent all calls to these methods, use the replacecall agent.

The Randoop replacecall agent transforms the code under test, replacing each call to an undesired method by a call to some other method. To use it:

By default, the agent makes the following replacements:

In addition, Randoop treats replaced methods as if the methods were given as arguments to --omitmethods. This prevents Randoop from creating tests with direct calls to them.

Replacement exclusions

There are some classes to which the replacecall agent should not be applied.

If you are using the agent in a build environment such as Ant, Maven, or Gradle, it is necessary to exclude the build environment classes. For instance, when running the agent on generated tests in a Gradle project using the Java plugin, the replacement of System.exit will affect the behavior of the build, and we should exclude the Gradle packages from being transformed. To exclude these packages, place the package names in an exclusion file gradle-exclusions.txt


and run the agent with the --dont-transform command-line option and the filename as its argument:


The exclusion file is a list of package names — one package per line. (Blank lines and //-style comments are ignored.) If a package name does not end with a period, one will be added. The package name is used to match the prefix of the fully-qualified classname.

For a different environment, you can determine what classes are being affected by running the agent with the --debug flag


and looking in the file transform-log.txt to see which methods are being transformed.

By default, the agent does not modify Randoop, the agent itself, or JUnit classes. So, by default, the agent will not make replacements in Randoop and Randoop Java agent classes, classes in certain subpackages of com.sun.tools and com.sun.source, or in JUnit or Hamcrest classes. Also, to improve performance, the agent does not make replacements in boot-loaded classes with the exception of packages in java.awt and javax.swing.

Defining replacements

To define your own method replacements, you will write mock implementations (example) and you will specify the replacements in a replacement file (example).

Give the replacement file to the replacecall agent using the --replacement-file argument:


A replacement method must be static and non-private. If the original method is static, the replacement method should have the identical signature. If the original method is non-static, then the replacement method should have an additional first parameter whose type should be the containing class of the original method. For instance, a call to java.awt.Component.show() would be replaced by a call to a method with the signature show(java.awt.Component). An example mock class is randoop.mock.java.awt.Component.

To simplify your replacement file and keep your mock classes organized, it is recommended that you choose a package prefix and keep the package structure of the original classes. The example file uses randoop.mock as the prefix, so the fully-qualified name of the JOptionPane mock class is randoop.mock.javax.swing.JOptionPane. (You may need to exclude this package if any of the replacement methods use otherwise replaced methods.)

Randoop comes with default replacements, which are in sub-packages of randoop.mock. You should not use the randoop.mock prefix for your own mock classes.

Keep the following in mind when defining replacements:

Implementing replacement classes

Sometimes, a replacement for an instance method needs to use a different object than the receiver (it might have the same or a different type than the receiver). In this case, your replacement class should maintain a map from instances of the original receiver to instances that will be used in the replacement. An example appears in {@code randoop.mock.java.util.MockRandom}.

Specifying expected code behavior (pre- and post-conditions)

You can provide a specification of the expected behavior of code. Randoop will use the specification to better classify method calls as error-revealing, expected behavior, or invalid.

You can specify both the behavior of methods and constructors, and representation invariants for classes.

Method specifications

A specification of a method (including constructors) indicates the circumstances when the method can be called, and how it should behave when called.

The --specifications=file command-line option names a file containing the method specifications. The file format is a JSON list with elements that are the serialized form of the OperationSpecification class. Randoop comes with an example specification file, which provides specifications for an example class net.Connection.

A method specification consists of the method signature, parameter names, a list of pre-conditions, a list of (normal) post-conditions, and a list of throws-conditions (exceptional post-conditions):

    "operation": method-signature,
    "identifiers": formal-parameters,
    "pre-conditions": pre-condition-list,
    "post-conditions": normal-post-condition-list,
    "throws-conditions": throws-condition-list

where only the operation field is required.

Method signatures

A method signature uniquely describes a method. It consists of the declaring class, method name, and parameter types. Types are given in the format used by Class.forName(); for example, member classes are set off with "$", as in pkg.Class$InnerClass.

The method net.Connection.send(int) is represented as

    "classname": "net.Connection",
    "name": "send",
    "parameterTypes": [ "int" ]

Formal parameter names

The conditions in a specification use identifiers, or variable names, to refer to formal parameters and the result. This part of the specification gives the names. For example:

    "parameters": [ "coded" ],
    "receiverName": "target",
    "returnName": "returnValue"

indicates that the single formal parameter of the method is referred to by the name coded. The receiver name may be omitted (default: receiver), and the return value name may be omitted (default: result).

Specifying conditions

The specification contains 3 lists of conditions, one for each different type of condition.


A pre-condition indicates a requirement that must be satisfied by the caller. Its guard is a Boolean expression that is evaluated in the pre-state — that is, before a method starts to execute. If the caller does not satisfy the pre-condition, then the call is invalid.

To specify that the argument to a call to net.Connection.send(int) must be positive, you could write the condition

      "description": "the code must be positive",
      "guard": "coded > 0"

The condition has an optional field description that Randoop uses in comments in the generated tests.

To require that the Connection is open and the code is non-negative, you could equivalently write a single pre-condition receiver.isOpen() && code > 0 or two pre-conditions receiver.isOpen() and code > 0.

(Normal) Post-condition

A post-condition indicates a requirement that must be satisfied by the method implementation. A normal postcondition indicates what values the method returns. If the caller satisfies the method's pre-condition and the method returns normally (without throwing an exception), then the post-condition must be satisfied or else the method is buggy.

You specify a normal post-condition as guard-property pair. (A property is given using the same format as a guard, but a property is evaluated in the post-state — that is, after a method has finished executing.) In the example file, the method net.Connection.receive() has the post-condition:

      "guard": "true",
      "property": "result >= 0",
      "description": "returns non-negative received value"

The guards of post-conditions are evaluated in the order that the post-conditions are listed. The list

        "guard": "!receiver.isOpen()",
        "property": "result == -1",
        "description": "if this is not open, return -1"
        "guard": "true",
        "property": "result >= 0",
        "description": "otherwise, returns non-negative received value"

indicates that the error-case should be tested first, followed by the general case that has the vacuous guard true.


A post-condition indicates requirements that must be satisfied by the method implementation. A throws-condition is a post-condition that indicates an exception the method is expected to throw. If a call satisfies the guard of a throws-condition, the exception must be thrown or the method is buggy.

You specify a throws-condition as a guard-exception pair. If a guard is satisfied, then the method is supposed to throw an exception of the named class, or a subclass of the named class. From the example file, the net.Connection.open() method has the throws-condition

        "guard": "receiver.isOpen()",
        "exception": "java.lang.IllegalStateException",
        "description": "throws IllegalStateException if the connection is already open"

Similar to normal post-conditions, throws-conditions are given in a list. The list, however, is treated as a set (meaning unordered), and may contain duplicate guards and exceptions. A method whose behavior is specified as

          "guard": "a == null",
          "exception": "java.lang.NullPointerException",
          "description": "throws NullPointerException if a is null"
          "guard": "i < 0",
          "exception": "java.lang.IndexOutOfBoundsException",
          "description": "throws IndexOutOfBoundsException if i is negative"

is considered as correct if, when called with a==null and i==-1, it throws either of the specified exceptions.

Evaluation rules

Randoop classifies a test sequence according to its last method call.

  1. If the guard of any pre-condition is violated before the call, then Randoop classifies the method call as invalid and the sequence is discarded.
  2. If the guard of any throws-condition is satisfied before the call, the exception from the condition is added to a set expected of expected exceptions. If the call throws a checked exception, the sequence is classified as having expected behavior if the thrown exception is a member of the set expected, or is a subclass of a member of expected, and is otherwise classified as error-revealing. If the expected set is non-empty and the call terminates normally, the sequence is classified as error-revealing.
  3. If the guard of a normal-precondition is satisfied before the call, the property for the first satisfied guard is checked after the call. If the call terminates normally, the sequence is classified as having expected behavior if the property is satisfied. Otherwise, the sequence is classified as error-revealing.
  4. If the method is inherited, the method call is also checked against all specifications of overridden methods in the inheritance chain. If the call is invalid by any specification, the sequence is classified as invalid. If the call is error-revealing by any specification, the sequence is classified as error-revealing. Otherwise, all specifications are satisfied and the sequence has expected behavior.
  5. Otherwise, the sequence is classified according to the rules of classifying tests.

Randoop does no check of whether the guards are exhaustive or disjoint.

Randoop ships with specifications for parts of the JDK, and these specifications are used by default, unless you disable command-line option --use-jdk-specifications.

Specifying representation invariant methods (such as checkRep)

The "representation invariant" of a class is a property that all instances of the class must satisfy. For example, a field might contain a non-negative number, or two fields might contain lists of the same length. Sometimes, a program contains a method that checks the rep invariant, for debugging purposes. If the rep invariant is ever violated, the program has a bug.

By using the @CheckRep annotation, you can tell Randoop which methods in your classes under test are rep invariant methods. Randoop will call these methods; if the method ever fails, Randoop outputs the test as a failing test.

A method annotated with @CheckRep must have one of two allowed signatures:

Instrumenting classes for filtering tests on covered-classes

Randoop generates tests that use methods, constructors, and fields of classes given using the --testclass and --classlist options. Any given test may involve one or more of the classes.

The --require-covered-classes option makes Randoop discard any test that doesn't use at least one of the specified classes.

Suppose you only want to test class A, but its methods takes arguments of type B and C. If you supply only A as a test class, Randoop cannot create objects of type B and C, so Randoop may be unable to call class A. Instead, you can supply A, B, and C as test classes, but instruct Randoop to only output the classes that use A.

Use of the --require-covered-classes option is unusual: you probably care about testing your entire application rather than just part of it. The option makes Randoop less efficient, because Randoop creates but discards tests.

For the --require-covered-classes option to work, the classes must be instrumented so Randoop can determine when they are covered. To instrument classes for this filter, also use the covered-class-4.0.0.jar by giving the following command-line option for java


This will instrument any non-abstract constructor and method of loaded classes so that the use of a class can be detected. (The instrumentation is not done on any JDK or JUnit classes.) (The covered-class-4.0.0.jar file can be found in the unzipped distribution archive, or can be downloaded from the latest release.)

Specifying additional primitive values

By default, Randoop uses the following pool of primitive values as inputs to methods:

This is the initial set of seed values used by Randoop: in addition to the above values, any primitive value returned at runtime from a method call under test may be later used as input to another method call.

There are two ways of specifying additional seed values to Randoop.


Randoop is deterministic: if you run Randoop twice, Randoop should output the same test suite.

This section gives several reasons why Randoop may produce different output when you run it twice, and tells you how to debug the problem. If after investigating these possibilities, you find that Randoop is nondeterministic because of a bug in Randoop, please report it so that we can investigate and correct the problem.

If you want Randoop to produce different test suites, supply different values for the --randomseed command-line argument.

Changes to the program or its environment

You changed the program or its environment.

Changing the JDK counts as a change to the program. For example, consider a class

public class Corner extends JComponent { ... }

In JDK 7, Corner's getAccessibleContext method is inherited from JComponent. In JDK 8, Corner's getAccessibleContext method is inherited from java.awt.Component. Depending on the JDK in use, Randoop discovers a different set of operations, because Corner.class.getMethods() returns different results depending on the JDK. Given different sets, Randoop may make different choices and produce different tests.

There can be implementation changes even within a single JDK major version, such as two different versions of JDK 8. These differences can lead to different Randoop outputs, or to tests that passed on one JDK implementation but fail on another.

Command-line arguments enable nondeterminism

You instructed Randoop to behave nondeterministically, for example by specifying a timeout. To forbid use of such options, pass the --deterministic command-line argument.

Nondeterministic program under test

The program under test is nondeterministic. This can cause Randoop to produce different tests on different runs, but more seriously, it can cause Randoop to produce flaky tests that sometimes pass and sometimes fail.

To determine which method is nondeterministic, run Randoop with the --log and --selection-log command-line options. Compare the logs across runs, and the first place that they differ indicates where the program is nondeterministic.

If the program under test is nondeterministic, you have several options:

  1. Change the program to be deterministic. In addition to making the program more testable, this is likely to make the program easier to debug. Here are some ways to eliminate nondeterminism in a sequential Java program:
  2. Transform the program at run time to make changes like the ones above, replacing the method's behavior with a different implementation (such as a deterministic one or a mock). You can do this by using Randoop's replacecall agent.
  3. Prevent Randoop from directly calling problematic methods. Use the --omitmethods or --omitmethods-list command-line argument.
  4. Instruct Randoop to attempt to discard or repair any flaky tests that it generates. Use the --ignore-flaky-tests command-line argument. This is a last resort because Randoop may output a small test suite due to discarding many of the generated tests. When possible, is better to avoid generating flaky tests rather than discarding them after generating them.

You can also see the techniques suggested in the paper Scaling Up Automated Test Generation.

Getting help

You can get help on Randoop in the following ways:


Randoop does not run

If the command to run Randoop has the output

Cannot find or load randoop.main.Main

then Randoop is not properly included in the classpath. One possibility is that you have included the Randoop jar file in the path specified by the CLASSPATH variable, but have given the location of the classes-under-test with the -classpath option at the command line. In this case, the java command will only use the classpath from -classpath.

It is recommended that both Randoop and the classes under test be included as the argument to the -classpath option to the java command. It only makes sense to set CLASSPATH to run Randoop if the location of classes-under-test will be the same each time you run it. See the Java Tutorial on path and classpaths.

If the command to run Randoop has the output

Cannot find the Java compiler. Check that classpath includes tools.jar

then you have installed Java incorrectly. You perhaps have the JRE on your path before the JDK. Ensure that JAVA_HOME appears at the front of your path. If you run the Windows operating system, see the instructions for setting a custom Java path after installing JDK 8.

Randoop cannot find class-under-test

If when Randoop is run, something like the following is output

No class found for type name "example.ExampleClass"

then the classes under test are not in the classpath. The most likely case is that you have not included the path to these classes. Otherwise, the classpath is not given properly. In particular, if you are running Randoop on Windows, parts of the classpath should be separated by a semi-colon ";" instead of the colon ":" shown in the examples in this document. See the Java Tutorial on paths and classpaths for platform specific information on paths.

Randoop does not create enough tests, or creates no tests

if Randoop's generated tests do not cover some part of your code (or if Randoop outputs "No tests were created. No JUnit class created."), then you need to change Randoop's parameters and re-run it. Spending just a little bit of effort may have a big impact on the quality of Randoop's output.

Consider some specific method that Randoop does not call, and determine why Randoop does not call that method. Run Randoop with logging enabled. Here is an incomplete list of possible problems and solutions:

If Randoop gets stuck, it outputs "*** Randoop has spent over 10 seconds executing the following test.", but it does not output any tests. Randoop creates and executes one test at a time, so a long-running sequence will prevent Randoop from creating new tests. Two possible reasons for a long-running test are:

Randoop does not terminate

If Randoop does not terminate, that might be because Randoop is creating tests that do not terminate (or that execute for a very long time). Here are three ways to deal with the problem.

Randoop produces different output on different runs

See the manual section on nondeterminism.

Randoop stopped because of a flaky test

A test is flaky if it behaves differently on different executions.

There are two main reasons for flaky tests. One is global side effects: a Randoop-generated test might call a method that sets a static field or creates/deletes a file. (Calling Randoop on a method that side-effects local state, such as a field of the receiver or of an argument, is fine.) Another reason is nondeterminism: the program under test might behave differently on different runs. Such behaviors are not appropriate for in unit tests, so you should not call Randoop on such methods.

If Randoop encounters a flaky test, Randoop halts so that you can correct your invocation of Randoop.

If you find a flaky test that does not result from side effects or nondeterminism in the program, then please help us improve Randoop by submitting an issue via the issue tracker.

Randoop produces a regression test suite that does not pass

Randoop outputs regression tests that passed during test generation. If the regression tests fail when you run them, they are said to be flaky. This is due to nondeterminism in the program under test. See the manual section on nondeterminism.

Tests behave differently in isolation or when reordered

Another symptom of calling Randoop on methods that have global side effects, in addition to flaky tests, is dependent tests: tests that behave differently depending on whether some other test has already run. For instance, test B might succeed if run after test A, but fail if run on its own.

You can either always run Randoop's generated tests in the same order (without performing test selection, prioritization, or parallelization), or you can avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

Another possible symptom is that if you run Randoop's tests and then your original test suite, the original test suite might fail because of side effects performed by Randoop's tests. To avoid this problem, you can run Randoop's tests in a separate JVM, or run Randoop's tests last, or avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

Reporting a bug to the issue tracker

If you have discovered a problem with Randoop, we would like to fix it. Please submit an issue to the issue tracker.

You should include enough information to enable the Randoop developers to reproduce the problem, which will make it easier for them to diagnose the problem, fix Randoop, and verify the fix. This generally includes

Never submit just a screenshot or image of your computer screen. Instead, cut-and-paste the text of your command and its output. The text is easier to read, is searchable, and does not require transcribing from a screenshot in order to reproduce.

If you encountered the problem while using the Maven plug-in, then please reproduce the problem from the command line, to help the Randoop developers find the underlying bug (or to isolate the problem to the plug-in).


The following individuals have contributed code to Randoop: Ayman Abdelghany, Michael Ernst, Waylon Huang, Laura Inozemtseva, René Just, Peter Kalauskas, Ben Keller, Carlos Pacheco, Jeff Perkins, Mark Roberts, Sai Zhang.

The feedback of Randoop users has been very valuable.

If your name has been inadvertently omitted from this section, please let us know so we can correct the oversight.