Randoop logo

Randoop Manual

This is the manual for Randoop version 4.3.2, released January 8, 2023. The Randoop homepage is https://randoop.github.io/randoop/.

Contents:

Randoop.NET is a version of Randoop that works with Microsoft's .NET platform, such as the C# programming language.

There is a separate Randoop Developer's Manual.

Introduction

Writing tests is important, but difficult and time-consuming. Randoop automatically generates unit tests for Java classes. Randoop has had many successful applications, especially with library classes (such as java.util). Randoop is used at companies like ABB and Microsoft, and on open-source projects.

Here is a JUnit test case generated by Randoop that reveals an error in OpenJDK (comments added by hand): 


  // This test shows that the JDK collection classes
  // can create an object that is not equal to itself.
  @Test
  public static void test1() {

    LinkedList list = new LinkedList();
    Object o1 = new Object();
    list.addFirst(o1);

    // A TreeSet is an ordered collection. According to the API
    // documentation, this constructor call should throw a
    // ClassCastException because the list element is not Comparable. But
    // the constructor silently (and problematically) accepts the list.
    TreeSet t1 = new TreeSet(list);

    Set s1 = Collections.synchronizedSet(t1);

    // At this point, we have successfully created a set (s1)
    // that violations reflexivity of equality: it is not equal
    // to itself! This assertion fails at run time on OpenJDK.
    org.junit.Assert.assertEquals(s1, s1);
  }

Randoop outputs two kinds of tests:

Typical use of Randoop

Here is a typical way to use Randoop:

  1. If Randoop outputs any error-revealing tests, fix the underlying defects, then re-run Randoop and repeat until Randoop outputs no error-revealing tests.
  2. Add the regression tests to your project's test suite.
  3. Run the regression tests whenever you change your project. These tests will notify you of changes to the behavior of your program.
  4. If any test fails, minimize the test case and then investigate the failure. If you disregarded any test (or if you added new code that you would like to test), then re-run Randoop to generate a new regression test suite that replaces the old one.

The paper "Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs" gives additional tips for how to use Randoop-generated tests over the lifetime of a project.

A typical programmer will only ever examine very few Randoop tests — when they fail and reveal a bug or a regression failure — and then only minimized versions of them. A typical programmer will never modify Randoop tests by hand.

Installing Randoop

Randoop runs on a Java 8, Java 11, Java 17, or Java 19 JVM.

Download and unzip the file randoop-4.3.2.zip. This manual uses ${RANDOOP_PATH} to refer to the path of the unzipped archive, and ${RANDOOP_JAR} to refer to the location of randoop-all-4.3.2.jar within the unzipped archive.

Alternately, if you want to view or work with the source code, follow the instructions in the Getting Started section of the Randoop Developer's Manual.

You may wish to join the randoop-discuss@googlegroups.com mailing list so that you are notified of new releases.

Running Randoop

Run Randoop by invoking its main class randoop.main.Main: 

java randoop.main.Main command args...

Randoop supports three commands:

(On Windows, adjust the classpath, such as using semicolon instead of colon as the separator.)

Generating tests

By default, Randoop generates and then outputs two kinds of unit tests, written to separate files.

Other generated tests, classified as invalid, are discarded.

This section of the manual gives an example of how to run Randoop, then describes test classification, error-revealing tests, and regression tests, and finally lists Randoop's command-line options.

Example: Generating tests for java.util.Collections

Imagine we want to generate tests for the class java.util.Collections. (This example is a bit unrealistic! More likely you would want to generate tests for all of the JDK, or for parts of it like java.lang and java.util.)

Randoop's tests only use classes you specify. In order to effectively test Collections, you should probably also specify some helper classes, including classes that generate collections. For this example, we will add java.util.TreeSet to the mix.

Create a file myclasses.txt that lists the names of the classes under test: 

java.util.Collections
java.util.TreeSet

and invoke Randoop as follows: 

java -classpath ${RANDOOP_JAR} randoop.main.Main gentests --classlist=myclasses.txt --time-limit=60

After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following: 

Created file: my/home/directory/RegressionTest0.java
Created file: my/home/directory/RegressionTest.java
Created file: my/home/directory/ErrorTest0.java
Created file: my/home/directory/ErrorTest.java
done.

Randoop creates two different test suites, in classes ErrorTest and RegressionTest. Compile and run the tests. (The classpath should include the code under test, the generated tests, and JUnit files junit.jar and hamcrest-core.jar. Classes in java.util.* are always on the Java classpath, so the myclasspath part is not needed in this particular example, but it is shown because you will usually need to supply it.) 

export JUNITPATH=.../junit.jar:.../hamcrest-core.jar
javac -classpath .:$JUNITPATH ErrorTest*.java RegressionTest*.java -sourcepath .:path/to/files/under/test/
java -classpath .:$JUNITPATH:myclasspath org.junit.runner.JUnitCore ErrorTest
java -classpath .:$JUNITPATH:myclasspath org.junit.runner.JUnitCore RegressionTest

All of the tests in ErrorTest will fail, and all of the tests in RegressionTest will pass.

You also might want to minimize the error-revealing tests before running them: 

java -cp .:junit.jar:myclasspath:.../randoop/build/libs/randoop-all-4.3.2.jar \
  randoop.main.Main minimize --suitepath=ErrorTest0.java --suiteclasspath=.:junit-4.12.jar:myclasspath

Specifying methods, constructors, and fields that may appear in a test

You need to tell Randoop what code it may call. In the common case of running Randoop on an entire program or library, it is usually enough to run Randoop on the .jar file. If you want to generate tests for only specific classes, then you need to do more work.

Suppose you want to test method A.m. You should direct Randoop to call not just A.m, but also methods that create A.m's inputs. More concretely, if A.m's parameter type is B, then you should tell Randoop to also call methods or constructors that create objects of type B. When a test calls A.m, all of the methods that set up state or create argument values are also being implicitly tested.

This command gives you an over-estimate (because it includes method call types as well as method argument types): 

  jdeps -apionly -v -R -cp myclasspath MyClass.class \
    | grep -v '^[A-Za-z]' | sed -E 's/^.* -> ([^ ]+) .*$/\1/' | sort | uniq

You may want to adjust the command-line arguments. For example, you can pass a .jar file instead of a .class file. Or, you could remove -R (and possibly run it twice to get depth-2 instead of depth-1).

It is unusual to specify just one or a few classes under test. Usually, you want to test an entire application or library. Specifying too few classes under test can prevent Randoop from achieving good coverage.

Which methods Randoop will call

Randoop only invokes a method or constructor M if all of the following are true:

  1. M is declared in a class that is named by --testjar, --classlist, or --testclass, or M is a member named by --methodlist.
  2. M does not match any of the patterns given via --omit-methods or --omit-methods-file.
  3. All of the following are accessible from the package of the tests (set with --junit-package-name): M's class, M, and M's parameter types and return type.

A method or constructor M that satisfies the above conditions is called a "method under test". Randoop only calls methods under test; that is, these are the only methods that will appear in a test case.

Use raw type names such as List, not parameterized types such as List<String>, when specifying classes or methods.

Do not include methods that side-effect global state, for two reasons. (Methods that side-effect the state of their receiver or other arguments are fine.)

Classifying tests

Randoop generates many tests internally, and it classifies each generated test as error-revealing, expected behavior, or invalid. The classification depends primarily on whether the last statement throws an exception or violates a contract. You can also provide specifications of the methods under test to further help Randoop classify tests; see section "Specifying expected code behavior".

You can use command-line arguments to customize Randoop's rules for classifying a test that throws an exception.

By default, a thrown exception is not considered to indicate an error in the code under test. (Equivalently, none of the above command-line arguments defaults to ERROR.) This is because a method may have a precondition that is unknown to Randoop. For example, a binary search implementation might require that its input is sorted. Randoop should not mark the method as buggy just because it throws an exception when Randoop passes it an ill-formed unsorted array. Another example is passing null to a method that requires a non-null argument, which might result in a NullPointerException, AssertionError, or other exception. Randoop has no way of knowing each method's precondition. In order to avoid outputting an "error-revealing test" that doesn't reveal an error but merely misuses the software under test, by default Randoop assumes that any exception thrown by a method is correct behavior in response to the values that it was passed.

Error-revealing tests

When Randoop includes a error-revealing test in file ErrorTest.java, the test shows that the code violates its specification or contract. For example, the test shown in the Introduction creates a TreeSet that violates reflexivity of equality: namely, for every object o, o.equals(o) should return true. The test reveals an error in the TreeSet constructor, which does not properly check its arguments (and possibly a second error in the TreeSet.equals method, which should always check if its argument is the same object as itself and return true in such a case).

The test in the Introduction is nicely succinct: it shows a small number of method calls leading up to the assertion violation. Because Randoop's generation is pseudo-random, the actual test revealing the error can be significantly longer, and contain many irrelevant calls that are not necessary to reveal the contract violation. You may wish to minimize the test case to ease debugging.

What kinds of errors does Randoop check for?

When Randoop calls a method that creates an object, Randoop verifies that the object is well-formed. Currently, Randoop checks for the following contracts:

Violation of any of these contracts is highly likely to indicate an error.

You can add additional contracts to Randoop, that will be checked for every object Randoop creates. Define a contract in directory randoop/src/main/java/randoop/contract/, and reference it in the constructor for OperationModel. You can also provide specifications of methods in the tested code to help Randoop classify tests; see section "Specifying expected code behavior".

Regression tests

The tests in the RegressionTest*.java files assert the current behavior of the code under test: values returned or exceptions thrown. After you change your code, you can run these regression tests, which will alert you if your code changes affect the external behavior of the classes.

Here is an example of a regression test for the TreeSet class: 

// This test passes when executed
@Test
public void test10() throws Throwable {

  java.util.TreeSet var0 = new java.util.TreeSet();
  java.lang.Short var1 = new java.lang.Short((short)100);
  boolean var2 = var0.contains(var1);

  // Regression assertion (captures the current behavior of the code)
  org.junit.Assert.assertEquals(false, var2);
}

This test would fail if you introduced an error that caused contains to return true on an empty set.

A regression test can also assert what exception is currently thrown, and it will fail if the code is changed to no longer throw that exception.

Regression test failures

If a regression test fails even though you have not changed your program, then the test is flaky.

If a regression test fails after you change your program, there are three possible causes. You need to debug the test failure to determine which is the cause, and then take the appropriate action.

Usually, a change to your program causes many test failures, not just one. This is because Randoop generates many tests for each part of your program's functionality. Don't panic! In most cases, all of the failures have the same cause.

Verifying the cause of regression test failures

You may wish to verify the cause of the failures. A way to do this is to change the part of the program, that you suspect is the cause, back to its previous behavior. Once you are satisfied that you understand the failures, you can discard the tests and have Randoop generate new ones. For example, suppose that you believe that all the test failures are caused by a change to method m. You can change m back to its previous implementation and run the tests. If they all succeed, you can change m back to its new implementation and re-generate the tests. (As an alternative to making a temporary change to your source code, you can use the replacecall feature of the Randoop agent to replace the new functionality by the old functionality.)

Stages of test generation, and limiting test generation

Here is how Randoop attempts to generate a test:

  1. Randoop selects a method to call. (This is called a "step".) The selected method will be the last method call in the new test.
  2. Randoop chooses arguments to the method call, from among values that have been computed by previous tests.
    If Randoop cannot find values that are different from those in every previous call to the method, then Randoop starts over choosing arguments.
    Otherwise, Randoop has created a test candidate — a new, unique sequence of operations that ends with the selected method. The candidate's code is the concatenation of the code to produce the arguments (except that for primitives, just the value itself is used), followed by the method call.
  3. Randoop executes the test candidate and classifies its behavior. If its behavior is invalid, then Randoop discards it and starts over.
  4. Randoop outputs error-revealing and regression tests, in separate suites.

The user can provide limits on any part of the test generation process, or on Randoop's run time. Test generation stops when any of the following occurs (this is controlled by method AbstractGenerator.shouldStop):

Randoop does not optimize to generate the best suite within these limits. Rather, Randoop just halts when a limit is reached.

Furthermore, see the command-line options for threading for additional per-test time limits.

Minimizing a failing JUnit test suite

Minimized test cases are usually easier to diagnose. Randoop includes a minimizer that, given a JUnit test suite, minimizes each failing test case. Each minimized test fails in the same way as the original, having the same failing assertions with the same stack trace. The minimizer works by removing and simplifying methods and statements in its input file, without changing the test suite's failure behavior.

There are two ways to use the minimizer:

For both ways to use the minimizer, see the documentation for the command-line options for failing test suite minimization.

The rest of this section discusses Randoop's minimize command.

To minimize all failing tests in ErrorTest1.java, run Randoop as follows: 

java -cp randoop-all-4.3.2.jar randoop.main.Main minimize \
  --suitepath=ErrorTest0.java --suiteclasspath=junit.jar:commons-lang.jar --testsuitetimeout=30

The minimizer produces a Java file named InputTestSuiteMinimized.java where InputTestSuite.java is the file being minimized.

The minimizer creates a new output file, rather than modifying the original test suite in place, because sometimes the minimization loses important information. For example, when a test case fails due to a NullPointerException, the minimizer might replace the right hand side of an expression with the value null.

Customizing Randoop's behavior to your application

If you run Randoop without any command-line arguments or customizations to your environment, it will produce helpful tests. However, Randoop will produce much more helpful tests if you spend a little bit of time to tell Randoop about your application, and that is how we recommend using Randoop.

Command-line options

[+] means option can be specified multiple times

A "binary name" is like a fully-qualified name, but uses "$" to separate nested classes from their containing classes, rather than ".".
A Randoop "fully-qualified signature" is

where package-name is a period-separated list of identifiers, and argument-list is a comma-separated list of fully-qualified Java raw types, without spaces. Recall that a raw type does not contain type parameters. Compared to a regular type, it removes type parameter declarations such as <T>, and it and replaces any use of a type parameter by its upper bound.

Avoiding calls to specific methods: the replacecall agent

The code under test might make calls to methods that you do not want executed. Here are some examples.

It is not enough to use --omit-methods to prevent direct calls to these methods, because the methods may still be called indirectly. To prevent all calls to these methods (including direct calls), use the replacecall agent.

The Randoop replacecall agent transforms the code under test, replacing each call to an undesired method by a call to some other method. To use it:

Randoop uses a default set of replacements. You can define additional replacements.

Replacement exclusions

There are some classes that the replacecall agent should not transform. Suppose you are using Gradle to run Randoop or the generated tests. Randoop's replacement of System.exit will affect the behavior of the build.

The replacecall agent never modifies

(The exact rules appear in the Randoop source code in class CoveredClassTransformer.)

You can exclude additional packages from being transformed. Create an exclusion file replacecall-exclusions.txt containing, for example: 

mypackage.mysubpackage.
org.something.library.

and run the agent with the --dont-transform=filename command-line option: 

-javaagent:${RANDOOP_PATH}/replacecall-4.3.2.jar=--dont-transform=replacecall-exclusions.txt

The exclusion file is a list of package names, one package per line. (Blank lines and //-style comments are ignored.) If a package name does not end with a period, one will be added. The package name is used to match the prefix of the fully-qualified classname.

For diagnostic output (such as to see what classes are being transformed), run the agent with the --debug flag 

-javaagent:${RANDOOP_PATH}/replacecall-4.3.2.jar=--debug

Defining replacements

To define your own method replacements, you will write mock implementations (example) and you will specify the replacements in a replacement file (example).

Each line of a replacement file is whitespace-separated package names (replace all calls in all methods of all classes in the package and its subpackages), or whitespace-separated class names (replace all methods in the class), or whitespace-separated method signatures without return types (replace the first method by the second).

Give the replacement file to the replacecall agent using the --replacement-file argument: 

-javaagent:${RANDOOP_PATH}/replacecall-4.3.2.jar=--replacement-file=replacecall-replacements.txt

A replacement method must be static and non-private. (Therefore, replacement classes don't need to be instantiated.)

Please keep in mind that when you define a replacement, it can change the behavior of client code.

Tips about implementing replacement classes

To simplify your replacement file and keep your mock classes organized, you should choose a package prefix and keep the package structure of the original classes. Randoop's default replacements file uses randoop.mock as the prefix, so the binary name of the JOptionPane mock class is randoop.mock.javax.swing.JOptionPane. (You may need to exclude this package if any of the replacement methods use otherwise replaced methods.)

If the class containing the method you wish to replace is loaded by the bootstrap class loader, or if it is called from a class loaded by the bootstrap class loader, then your replacement classes must also be loaded by the bootstrap class loader. For Java 8 this list includes most of the Java runtime. For Java 9+, the new module system dramatically reduces the number of classes loaded by the bootstrap loader. See Module base for a list. If this condition applies to your replacement code (assume it is located in myreplace.jar):

Randoop's default replacements are in sub-packages of randoop.mock. You should not use the randoop.mock prefix for your own mock classes.

Sometimes, a replacement for an instance method needs to access an object that is different than the receiver (for example, it might have the same or a different type than the receiver). One way to manage this is for your replacement class to maintain a map from instances of the original receiver type to instances that will be used in the replacement method implementations.

Specifying expected code behavior

You can provide a specification of the expected behavior of code. Randoop will use the specification to better classify method calls as error-revealing, expected behavior, or invalid.

You can specify both the behavior of methods and constructors, and representation invariants for classes.

Method specifications

A specification of a method (including constructors) indicates the circumstances when the method can be called, and how it should behave when called.

The --specifications=file command-line option names a file containing the method specifications. The file format is a JSON list with elements that are the serialized form of the OperationSpecification class. You may also supply a .zip file containing such JSON files.

Randoop comes with an example specification file, which provides specifications for class net.Connection.

A method specification consists of the method signature, parameter names, a list of pre-conditions, a list of (normal) post-conditions, and a list of throws-conditions (exceptional post-conditions): 


  {
    "operation": method-signature,
    "identifiers": formal-parameters,
    "pre-conditions": pre-condition-list,
    "post-conditions": normal-post-condition-list,
    "throws-conditions": throws-condition-list
  }

where only the operation field is required.

Method signatures

A method signature uniquely describes a method. It consists of the declaring class, method name, and parameter types. Types are given in the format used by Class.forName(); for example, member classes are set off with "$", as in pkg.Class$InnerClass.

The method net.Connection.send(int) is represented as 


  {
    "classname": "net.Connection",
    "name": "send",
    "parameterTypes": [ "int" ]
  }

A constructor's name is the fully-qualified or simple class name. (The simple name contains no "." or "$".)

Formal parameter names

The conditions in a specification use identifiers, or variable names, to refer to formal parameters and the result. This part of the specification gives the names. For example: 


  {
    "parameters": [ "coded" ],
    "receiverName": "target",
    "returnName": "returnValue"
  }

indicates that the single formal parameter of the method is referred to by the name coded. The receiver name may be omitted (default: receiver), and the return value name may be omitted (default: result).

Specifying conditions

The specification contains 3 lists of conditions, one for each different type of condition.

Pre-condition

A pre-condition indicates a requirement that must be satisfied by the caller. Its guard is a Boolean expression that is evaluated in the pre-state — that is, before a method starts to execute. If the caller does not satisfy the pre-condition, then the call is invalid.

To specify that the argument to a call to net.Connection.send(int) must be positive, you could write the condition 


    {
      "description": "the code must be positive",
      "guard": "coded > 0"
    }

The condition has an optional field description that Randoop uses in comments in the generated tests.

To require that the Connection is open and the code is non-negative, you could equivalently write a single pre-condition receiver.isOpen() && code > 0 or two pre-conditions receiver.isOpen() and code > 0.

A pre-condition is used to filter values, not to generate them. Randoop uses a precondition to prune, not expand, its random search.

(Normal) Post-condition

A post-condition indicates a requirement that must be satisfied by the method implementation. A normal postcondition indicates what values the method returns. If the caller satisfies the method's pre-condition and the method returns normally (without throwing an exception), then the post-condition must be satisfied or else the method is buggy.

You specify a normal post-condition as guard-property pair. (A property is given using the same format as a guard, but a property is evaluated in the post-state — that is, after a method has finished executing.) In the example file, the method net.Connection.receive() has the post-condition: 


    {
      "guard": "true",
      "property": "result >= 0",
      "description": "returns non-negative received value"
    }

The guards of post-conditions are evaluated in the order that the post-conditions are listed. The list 


    [
      {
        "guard": "!receiver.isOpen()",
        "property": "result == -1",
        "description": "if this is not open, return -1"
      },
      {
        "guard": "true",
        "property": "result >= 0",
        "description": "otherwise, returns non-negative received value"
      }
    ]

indicates that the error-case should be tested first, followed by the general case that has the vacuous guard true.

Throws-condition

A post-condition indicates requirements that must be satisfied by the method implementation. A throws-condition is a post-condition that indicates an exception the method is expected to throw. If a call satisfies the guard of a throws-condition, the exception must be thrown or the method is buggy.

You specify a throws-condition as a guard-exception pair. If a guard is satisfied, then the method is supposed to throw an exception of the named class, or a subclass of the named class. From the example file, the net.Connection.open() method has the throws-condition 


      {
        "guard": "receiver.isOpen()",
        "exception": "java.lang.IllegalStateException",
        "description": "throws IllegalStateException if the connection is already open"
      }

Similar to normal post-conditions, throws-conditions are given in a list. The list, however, is treated as a set (meaning unordered), and may contain duplicate guards and exceptions. A method whose behavior is specified as 


      [
        {
          "guard": "a == null",
          "exception": "java.lang.NullPointerException",
          "description": "throws NullPointerException if a is null"
        },
        {
          "guard": "i < 0",
          "exception": "java.lang.IndexOutOfBoundsException",
          "description": "throws IndexOutOfBoundsException if i is negative"
        }
      ]

is considered as correct if, when called with a==null and i==-1, it throws either of the specified exceptions.

Evaluation rules

Randoop classifies a test sequence according to its last method call.

  1. If the guard of any pre-condition is violated before the call, then Randoop classifies the method call as invalid and the sequence is discarded.
  2. If the guard of any throws-condition is satisfied before the call, the exception from the condition is added to a set expected of expected exceptions. If the call throws a checked exception, the sequence is classified as having expected behavior if the thrown exception is a member of the set expected, or is a subclass of a member of expected, and is otherwise classified as error-revealing. If the expected set is non-empty and the call terminates normally, the sequence is classified as error-revealing.
  3. If the guard of a normal-precondition is satisfied before the call, the property for the first satisfied guard is checked after the call. If the call terminates normally, the sequence is classified as having expected behavior if the property is satisfied. Otherwise, the sequence is classified as error-revealing.
  4. If the method is inherited, the method call is also checked against all specifications of overridden methods in the inheritance chain. If the call is invalid by any specification, the sequence is classified as invalid. If the call is error-revealing by any specification, the sequence is classified as error-revealing. Otherwise, all specifications are satisfied and the sequence has expected behavior.
  5. Otherwise, the sequence is classified according to the rules of classifying tests.

Randoop does no check of whether the guards are exhaustive or disjoint.

Randoop ships with specifications for parts of the JDK, and these specifications are used by default, unless you disable command-line option --use-jdk-specifications.

Specifying representation invariant methods (such as checkRep)

The "representation invariant" of a class is a property that all instances of the class must satisfy. For example, a field might contain a non-negative number, or two fields might contain lists of the same length. Sometimes, a program contains a method that checks the rep invariant, for debugging purposes. If the rep invariant is ever violated, the program has a bug.

By using the @CheckRep annotation, you can tell Randoop which methods in your classes under test are rep invariant methods. Randoop will call these methods; if the method ever fails, Randoop outputs the test as a failing test.

A method annotated with @CheckRep must have one of two allowed signatures:

Testing a specific class or method

Randoop is designed to create tests for an entire project. This is usually what engineers want. In rare cases, you might want to create tests for a particular class or method. In general, it is better to run Randoop on your entire project, and Randoop will create useful, broad tests, including for the specific class or method. Randoop does not have a way to generate tests for a specific class or method. However, you can filter Randoop's output so that you only see some of its tests, using the --require-covered-classes or --require-classname-in-test command-line option.

If you want to test a particular class or method, do not attempt to use the --methodlist, --classlist, or similar command-line options. They restrict the methods that appear anywhere in a test, and are likely to lead to Randoop not producing any output.

Requiring tests to use a class: the covered-classes agent

Randoop generates tests that use methods, constructors, and fields of classes given using the --testjar, --classlist, and --testclass options. Any given test may involve one or more of the classes.

The --require-covered-classes=filename option makes Randoop discard any test that doesn't use at least one of the classes listed in the file.

Suppose you only want to test class A, but its methods takes arguments of type B and C. If you supply only A as a test class, Randoop cannot create objects of type B and C, so Randoop may be unable to call class A. Instead, you can supply A, B, and C as test classes, but instruct Randoop to only output the tests that use A.

Use of the --require-covered-classes option is unusual: you probably care about testing your entire application rather than just part of it. The option makes Randoop less efficient, because Randoop creates but discards tests.

For the --require-covered-classes option to work, the classes must be instrumented so Randoop can determine when they are covered. You can do so by giving the following command-line option for java 


-javaagent:${RANDOOP_PATH}/covered-class-4.3.2.jar

This will instrument any non-abstract constructor and method of loaded classes so that the use of a class can be detected. The instrumentation is not done on any JDK or JUnit classes. The covered-class-4.3.2.jar file can be found in the unzipped distribution archive, or can be downloaded from the latest release.

Specifying additional primitive values

Randoop chooses arguments to method calls from the result of any previous method call, and from a set of seed values. These are the built-in seed values:

There are two ways of specifying additional seed values to Randoop.

Randoop does not generate random string or primitive values. You can include, as a method under test, a method that creates values. Its output will be used as an input to methods in your code.

Nondeterminism

Randoop is deterministic: if you run Randoop twice using the --deterministic command-line argument, Randoop outputs the same tests in the same order.

If you want Randoop to produce different test suites, run Randoop twice supplying different values for the --randomseed command-line argument. This is the recommended practice, to yield an overall test suite with better coverage.

Finding the source of nondeterminism

If Randoop produces a different output when you run it twice with the --deterministic command-line argument, then there is some source of nondeterminism.

If Randoop produces flaky tests and --flaky-test-behavior is not set to HALT, Randoop will output a list of methods suspected to be the cause of nondeterminism in decreasing order of likelihood. You should investigate these methods, fix nondeterministic ones or exclude them via --omit-methods or --omit-methods-file command-line argument, and re-run Randoop until flaky tests are not generated. If you discover a nondeterministic method, let the Randoop maintainers know so we can include it in Randoop's default omitmethods-defaults.txt or JDK-nondet-methods.txt.

The remainder of this section gives several reasons you may discover.

Nondeterministic program under test

If the program under test is nondeterministic, Randoop may produce different tests on different runs. More seriously, Randoop may produce flaky tests that sometimes pass and sometimes fail.

If the program under test is nondeterministic, you have several options:

  1. Change the program to be deterministic. In addition to making the program more testable, this is likely to make the program easier to debug. Here are some ways to eliminate nondeterminism in a sequential Java program:
  2. Transform the program at run time to make changes like the ones above, replacing the method's behavior with a different implementation (such as a deterministic one or a mock). You can do this by using Randoop's replacecall agent.
  3. Prevent Randoop from directly calling problematic methods. Use the --omit-methods or --omit-methods-file command-line argument.
  4. Instruct Randoop to attempt to discard or repair any flaky tests that it generates. Use the --flaky-test-behavior command-line argument. This is a last resort because Randoop may output a small test suite due to discarding many of the generated tests. When possible, is better to avoid generating flaky tests rather than discarding them after generating them.

You can also see the techniques suggested in the paper Scaling Up Automated Test Generation.

Changes to the program or its environment

If the program or its environment changed between runs, Randoop may produce different test suites..

Changing the JDK counts as a change to the program. For example, consider a class 

public class Corner extends JComponent { ... }

In JDK 7, Corner's getAccessibleContext method is inherited from JComponent. In JDK 8, Corner's getAccessibleContext method is inherited from java.awt.Component. Depending on the JDK in use, Randoop discovers a different set of operations, because Corner.class.getMethods() returns different results depending on the JDK. Given different sets, Randoop may make different choices and produce different tests.

There can be implementation changes even within a single JDK major version, such as two different versions of JDK 8. These differences can lead to different Randoop outputs, or to tests that passed on one JDK implementation but fail on another.

Debugging nondeterminism in Randoop

If the above techniques don't help you, you can diagnose nondeterminism by running Randoop twice with the command-line options 

  --deterministic --log=randoop-log.txt --selection-log=selection-log.txt

Compare the logs across runs, and the first place that they differ will help you find the nondeterminism.

If you find that Randoop is nondeterministic because of a bug in Randoop, please report it so that we can investigate and correct the problem.

Getting help

You can get help on Randoop in the following ways:

Troubleshooting

Randoop does not run

If the command to run Randoop has the output 

Cannot find or load randoop.main.Main

then Randoop is not properly included in the classpath. One possibility is that you have included the Randoop jar file in the path specified by the CLASSPATH variable, but have given the location of the classes-under-test with the -classpath option at the command line. In this case, the java command will only use the classpath from -classpath.

It is recommended that both Randoop and the classes under test be included as the argument to the -classpath option to the java command. It only makes sense to set CLASSPATH to run Randoop if the location of classes-under-test will be the same each time you run it. See the Java Tutorial on path and classpaths.

If the command to run Randoop has the output 

Cannot find the Java compiler. Check that classpath includes tools.jar

then there is a problem with your Java installation. You perhaps have the JRE on your path before the JDK. Ensure that JAVA_HOME appears at the front of your path. If you run the Windows operating system, see the instructions for setting a custom Java path. Another possible cause of the problem is using an unsupported version of Java.

Randoop cannot find a class-under-test

If when Randoop is run, something like the following is output 

No class found for type name "example.ExampleClass"

Unable to load class "example.ExampleClass" due to exception: java.lang.ClassNotFoundException

MyClass was read from myjar.jar but was not found on classpath.  Ensure that myjar.jar is on the classpath.

then the classes under test are not in the classpath. The most likely case is that you have not included the path to these classes. Another possible cause is that you did not supply the classpath to Randoop properly.

As described in the "Running Randoop" section, a typical invocation is 

    java -classpath myclasspath:${RANDOOP_JAR} randoop.main.Main ...

If you are running Randoop on Windows, parts of the classpath should be separated by a semicolon ";" instead of the colon ":" shown in the examples in this document. See the Java Tutorial on paths and classpaths for platform-specific information on paths.

Randoop does not create enough tests, or creates no tests

if Randoop's generated tests do not cover some part of your code (or if Randoop outputs "No tests were created. No JUnit class created."), then you need to change Randoop's parameters and re-run it. Spending just a little bit of effort may have a big impact on the quality of Randoop's output.

Consider some specific method that Randoop does not call, and determine why Randoop does not call that method. Run Randoop with logging enabled. Here is an incomplete list of possible problems and solutions:

  1. The method is not a candidate to be called by Randoop. Randoop creates a "model", or a list of methods that it might call. This list appears in the Randoop logs after the text "Operations:". If the method is not in that list, determine why. Two common reasons are that the method (or the class that defines it) is not public or that it is excluded by an --omit-methods or --omit-methods-file argument (or the default omissions).
  2. Randoop never tries to call the method. That is, Randoop never randomly selects this method, from among all the methods in the model. You should run Randoop with a larger timeout. Alternately, you could run Randoop with fewer classes, or exclude some classes or methods from consideration. You might also avoid using --usethreads, which can make Randoop run very slowly.
  3. Randoop tries to call the method, but is unable to create inputs (arguments to the method). You should give Randoop more classes to analyze, namely ones that create the objects that the method needs as input.
  4. Randoop calls the method, but the inputs it uses are not valid/useful inputs. You should investigate why Randoop is unable to create the needed inputs. Maybe you need to give Randoop more classes to analyze (as in the previous bullet point) or provide additional primitive values.

If Randoop gets stuck, it outputs "*** Randoop has spent over 10 seconds executing the following test.", but it does not output any tests. Randoop creates and executes one test at a time, so a long-running sequence will prevent Randoop from creating new tests. Two possible reasons for a long-running test are:

Randoop does not terminate

When Randoop does not terminate, that is usually because Randoop is creating tests that do not terminate (or that execute for a very long time). Here are three things you can do.

If you cannot fix the problem that causes the long execution, you might want to exclude the method from testing.

Randoop produces different output on different runs

See the manual section on nondeterminism.

Randoop generated flaky tests

A test is flaky if it behaves differently on different executions.

There are two main reasons for flaky tests. One is global side effects: a Randoop-generated test might call a method that sets a static field or creates/deletes a file. (Calling Randoop on a method that side-effects local state, such as a field of the receiver or of an argument, is fine.) Another reason is nondeterminism: the program under test might behave differently on different runs. Such behaviors are not appropriate for in unit tests, so you should not call Randoop on such methods.

If Randoop encounters a flaky test, Randoop halts so that you can correct your invocation of Randoop. Another section of the manual gives steps for determining the source of nondeterminism.

If you find a flaky test that does not result from side effects or nondeterminism in the program, then please help us improve Randoop by submitting an issue via the issue tracker.

Randoop produces a regression test suite that does not pass

Randoop outputs regression tests that passed during test generation. If the regression tests fail when you run them, they are said to be flaky. This is due to nondeterminism in the program under test. See the manual section on nondeterminism.

Tests behave differently in isolation or when reordered

Another symptom of calling Randoop on methods that have global side effects, in addition to flaky tests, is dependent tests: tests that behave differently depending on whether some other test has already run. For instance, test B might succeed if run after test A, but fail if run on its own.

You have two choices. (1) Always run Randoop's generated tests in the same order (without performing test selection, prioritization, or parallelization). (2) Avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

Another possible symptom is that if you run Randoop's tests and then your original test suite, the original test suite might fail because of side effects performed by Randoop's tests. To avoid this problem, you can run Randoop's tests in a separate JVM, or run Randoop's tests last, or avoid calling Randoop on methods with global side effects as described in the section on flaky tests.

A fatal error is detected by the Java Runtime

If the command to run Randoop has output similar to 

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f925c9f2320, pid=28883, tid=0x00007f922dfc9700
#

This indicates that a segmentation fault has occured within the Java Runtime Environment. This almost always is caused by a bug in the JRE --- it should never crash. There are two potential ways to work around this problem:

  1. Try using a newer version of Java if one is available. We have seen this problem most often with JDK 8.
  2. When this fault occurs, the Java Runtime will produce a core dump. Try looking at the stack trace in the core dump to determine the test class that led to the fault, then use one of the Randoop options to remove this class from consideration by Randoop.

Reporting a bug to the issue tracker

If you have discovered a problem with Randoop, we would like to fix it. Please submit an issue to the issue tracker. (Before doing so, search the open issues and the manual, especially the Troubleshooting section.)

You should include enough information to enable the Randoop developers to reproduce the problem, which will make it easier for them to diagnose the problem, fix Randoop, and verify the fix. This generally includes

Never submit just a screenshot or image of your computer screen. Instead, cut-and-paste the text of your command and its output. The text is easier to read, is searchable, and does not require transcribing from a screenshot in order to reproduce.

If you encountered the problem while using the Maven plug-in or running a command within an IDE, then please reproduce the problem from the command line, to help the Randoop developers find the underlying bug (or to isolate the problem to the plug-in or IDE configuration).

Credits

The following individuals have contributed code to Randoop: Alessandra Gorla, Ayman Abdelghany, Ben Keller, Carlos Pacheco, Casey Xing, Ivan Kocherhin, Jeff Perkins, Laura Inozemtseva, Lilia Tang, Mark Roberts, Martin Kellogg, Michael Ernst, Peter Kalauskas, René Just, Sai Zhang, Shuqi Lin, Sonion Lee, Suzanne Millstein, Thad Guidry, Waylon Huang.

The feedback of Randoop users has been very valuable.