Note

The CartesianProductTest extension has undergone significant changes in 1.6.0. This included renaming it to @CartesianTest and moving it into a new package. This here is the old variant, which was deprecated in 1.6.0 and removed in 2.0. For the new version, see here.

From Wikipedia:

In mathematics, specifically set theory, the Cartesian product of two sets A and B, denoted A × B, is the set of all ordered pairs (a, b) where a is in A and b is in B. In terms of set-builder notation, that is A × B = {(a,b) | a ∈ A and b ∈ B} [...] One can similarly define the Cartesian product of n sets, also known as an n-fold Cartesian product, which can be represented by an n-dimensional array, where each element is an n-tuple.

What does all this mean?

The Cartesian product of sets is all the possible combinations where you take a single element from each set. If you have two sets, { 1, 2 } and { 3, 4 }, their cartesian product is { { 1, 3 }, { 1, 4 }, { 2, 3 }, { 2, 4 } }.

Sometimes it’s useful to test all possible combinations of parameter sets. Normally, this results in a lot of written test data parameters. For a more comfortable way you may use the @CartesianProductTest extension. The extension takes the test data parameter values and runs the test for every possible combination of them.

Basic Use

@CartesianProductTest is used instead of @Test or other such annotations (e.g. @RepeatedTest).

You can supply test parameters to @CartesianProductTest in multiple ways.

Specifying more than one kind of parameter source (e.g.: both annotating your test method and having a static factory) does not work and will throw an ExtensionConfigurationException.

Our earlier example with { 1, 2 } and { 3, 4 }, would look like this:

@CartesianProductTest
@CartesianValueSource(ints = { 1, 2 })
@CartesianValueSource(ints = { 3, 4 })
void myCartesianTestMethod(int x, int y) {
    // passing test code
}

@CartesianProductTest works with parameters injected by JUnit automatically (e.g.: TestReporter). You can read about auto-injected parameters here.

Just like the mathematical Cartesian product, @CartesianProductTest works with sets. Duplicate elements get removed automatically. If your input is { 1, 1, 3 } and { 2, 2 } the extension will consider their Cartesian product { { 1, 2 }, { 3, 2 } }. Otherwise, the test would run with the same parameters multiple times. If you need to pass the same parameters multiple times, you might want to look into repeated tests.

Supplying CartesianProductTest with a String[]

If all your test parameters are strings, you can supply all input parameters simultaneously by giving a string array value to @CartesianProductTest. This value is the input for all parameters. The test will try every combination of its elements.

@CartesianProductTest({ "0", "1" })
void threeBits(String a, String b, String c) {
    // passing test code
}

The test threeBits is executed exactly eight times, because all three input parameters can have the values "0" or "1". @CartesianProductTest tests for all input combinations, that’s 2 × 2 × 2, so eight tests in total.

To demonstrate with a table:

# of test value of a value of b value of c

1st test

"0"

"0"

"0"

2nd test

"0"

"0"

"1"

3rd test

"0"

"1"

"0"

4th test

"0"

"1"

"1"

5th test

"1"

"0"

"0"

6th test

"1"

"0"

"1"

7th test

"1"

"1"

"0"

8th test

"1"

"1"

"1"

Annotating your test method

@CartesianValueSource

If you don’t only supply string values to your test method like in the example of the previous section, you can annotate your method with @CartesianValueSource. @CartesianValueSource is used to define the possible inputs of a single test parameter - as annotations are listed top-to-bottom, they provide parameter values left-to-right. The test will try every combination those values can have.

@CartesianProductTest
@CartesianValueSource(ints = { 1, 2, 4 })
@CartesianValueSource(strings = { "A", "B" })
void testIntChars(int number, String character) {
    // passing test code
}

This annotation might look familiar - it mimics JUnit’s @ValueSource, except @CartesianValueSource is repeatable. It also does NOT work with @ParameterizedTest.

The test testIntChars is executed exactly six times. The first parameter can have any of the three values 1, 2 or 4. The second parameter can have any of the two values "A" or "B". @CartesianProductTest tests for all input combinations, that’s 3 × 2, so six tests in total.

To demonstrate with a table:

# of test value of number value of character

1st test

1

"A"

2nd test

1

"B"

3rd test

2

"A"

4th test

2

"B"

5th test

4

"A"

6th test

4

"B"

@CartesianEnumSource

@CartesianEnumSource provides a convenient way to use Enum constants.

@CartesianProductTest
@CartesianEnumSource(ChronoUnit.class)
void testWithCartesianEnumSource(TemporalUnit unit) {
    assertThat(unit).isNotNull();
}

Like @CartesianValueSource, @CartesianEnumSource is an annotation that might look familiar - it mimics JUnit’s @EnumSource, except @CartesianEnumSource is repeatable. It also does NOT work with @ParameterizedTest.

The annotation’s value attribute is optional. When omitted, the declared type of the parameter of the @CartesianProductTest method, which has the same relative index of the annotation, is used. The test will fail if it does not reference an enum type. Thus, the value attribute is required in the above example because the method parameter is declared as TemporalUnit, i.e. the interface implemented by ChronoUnit, which isn’t an enum type. Changing the method parameter type to ChronoUnit allows you to omit the explicit enum type from the annotation as follows.

@CartesianProductTest
@CartesianEnumSource
void testWithCartesianEnumSourceWithAutoDetection(ChronoUnit unit) {
    assertThat(unit).isNotNull();
}

As the above example has only one annotation, the type of the first parameter is used. The automatic detection of the enum type also works in cases with several @CartesianEnumSource annotations, even when mixed with other annotation supported by @CartesianProductTest.

@CartesianProductTest
@IntRangeSource(from = 0, to = 2)
@CartesianEnumSource
@CartesianEnumSource
@CartesianValueSource(longs = { 2, 3 })
void testWithCartesianEnumSourceMixedWithOtherAnnotations(int i, TestEnum e1, AnotherTestEnum e2, long l) {
    assertThat(i).isNotNull();
    assertThat(e1).isNotNull();
    assertThat(e2).isNotNull();
    assertThat(l).isNotNull();
}

The annotation provides an optional names attribute that lets you specify which constants shall be used, like in the following example. If omitted, all constants will be used.

@CartesianProductTest
@CartesianEnumSource(names = { "DAYS", "HOURS" })
void testWithCartesianEnumSourceInclude(ChronoUnit unit) {
    assertThat(EnumSet.of(ChronoUnit.DAYS, ChronoUnit.HOURS)).contains(unit);
}

The annotation also provides an optional mode attribute that enables fine-grained control over which constants are passed to the test method. For example, you can exclude names from the enum constant pool or specify regular expressions as in the following examples.

@CartesianProductTest
@CartesianEnumSource(mode = EXCLUDE, names = { "ERAS", "FOREVER" })
void testWithCartesianEnumSourceExclude(ChronoUnit unit) {
    assertThat(EnumSet.of(ChronoUnit.ERAS, ChronoUnit.FOREVER)).doesNotContain(unit);
}
@CartesianProductTest
@CartesianEnumSource(mode = MATCH_ALL, names = "^.*DAYS$")
void testWithCartesianEnumSourceRegex(ChronoUnit unit) {
    assertThat(unit.name()).endsWith("DAYS");
}

The example below shows how to use @CartesianEnumSource with two Enum types.

enum MyEnum {
    ONE, TWO, THREE
}

enum AnotherEnum {
    ALPHA, BETA, GAMMA, DELTA
}

@CartesianProductTest
@CartesianEnumSource(MyEnum.class)
@CartesianEnumSource(value = AnotherEnum.class, names = { "ALPHA", "DELTA" }, mode = Mode.EXCLUDE)
void testEnumValues(MyEnum myEnum, AnotherEnum anotherEnum) {
    // passing test code
}

The test testEnumValues is executed exactly six times. The first parameter can have any of the three constants ONE, TWO or THREE. The second parameter can have any of the two constants BETA or GAMMA (note the EXCLUDE mode applied to the other two constants). @CartesianProductTest tests for all input combinations, that’s 3 × 2, so six tests in total.

To demonstrate with a table:

# of test value of myEnum value of anotherEnum

1st test

ONE

BETA

2nd test

ONE

GAMMA

3rd test

TWO

BETA

4th test

TWO

GAMMA

5th test

THREE

BETA

6th test

THREE

GAMMA

@CartesianEnumSource can also be combined with other supported annotations, like @CartesianValueSource:

enum MyEnum {
    ONE, TWO, THREE
}

@CartesianProductTest
@CartesianValueSource(ints = { 1, 2, 3 })
@CartesianEnumSource(MyEnum.class)
void testEnumValues(int i, MyEnum myEnum) {
    // passing test code
}

Range Source annotations

Alternatively, you can annotate your test method with range source annotations. For this purpose only, range sources are repeatable. When you annotate your @CartesianProductTest with a range source, you specify the inputs to a single test parameter. This is not like @ParameterizedTest where you have to define all inputs in a single @ArgumentsSource.

@CartesianProductTest
@ShortRangeSource(from = 1, to = 3, step = 1)
@LongRangeSource(from = 0L, to = 2L, step = 1, closed = true)
void testShortAndLong(short s, long l) {
    // passing test code
}

Just like with @CartesianValueSource, annotations get assigned to parameters in order. The first annotation from the top gets assigned to the first parameter, the second annotation to the second parameter and so on.

The test testShortAndLong is executed exactly six times. The first parameter can have any of the two values 1, 2. The second parameter can have any of the three values 0L, 1L or 2L. @CartesianProductTest tests for all input combinations, that’s 2 × 3, so six tests in total.

To demonstrate with a table:

# of test value of s value of l

1st test

1

0L

2nd test

1

1L

3rd test

1

2L

4th test

2

0L

5th test

2

1L

6th test

2

2L

For more information, please see the separate documentation about range sources. You can combine range sources with @CartesianValueSource, it works as you’d expect:

@CartesianProductTest
@ShortRangeSource(from = 1, to = 3)
@CartesianValueSource(strings = { "text one", "text two" })
void mixAndMatchTest(short number, String text) {
    // passing test code
}

Writing a static factory method for the parameters

If your tests require special inputs that @CartesianValueSource is not able to supply, you can define a static factory method to supply your test parameters. By default, this method must have the same name as the test method, but you can specify a different name with the factory annotation parameter. Just like with JUnit’s @MethodSource, you can specify the factory method with its fully-qualified name (including the class), e.g. com.example.Class#factory. This method must return CartesianProductTest.Sets. CartesianProductTest.Sets is a helper class, specifically for creating sets for @CartesianProductTest. To create the test data, instantiate a new CartesianProductTest.Sets() then use the methods add() (with varargs arguments) or addAll() (with Iterable or Stream argument) to register the values for the parameters.

@CartesianProductTest
void nFold(String string, Class<?> clazz, TimeUnit unit) {
    // passing test code
}

static CartesianProductTest.Sets nFold() {
    return new CartesianProductTest.Sets()
        // e.g. with a `List<String> letters`,
        // you can use `addAll`:
        // .addAll(letters)
        .add("Alpha", "Omega")
        .add(Runnable.class, Cloneable.class, Predicate.class)
        .add(TimeUnit.DAYS, TimeUnit.HOURS);
}

The test nFold is executed exactly twelve times. The first parameter can have any of the two values "Alpha" or "Omega". The second parameter can have any of the three values Runnable.class, Cloneable.class or Predicate.class. The third parameter can have any of the two values TimeUnit.DAYS or TimeUnit.HOURS. @CartesianProductTest tests for all input combinations, that’s 2 × 3 × 2, so twelve tests in total.

To demonstrate with a table:

# of test value of string value of clazz value of unit

1st test

"Alpha"

Runnable.class

TimeUnit.DAYS

2nd test

"Alpha"

Runnable.class

TimeUnit.HOURS

3rd test

"Alpha"

Cloneable.class

TimeUnit.DAYS

4th test

"Alpha"

Cloneable.class

TimeUnit.HOURS

5th test

"Alpha"

Predicate.class

TimeUnit.DAYS

6th test

"Alpha"

Predicate.class

TimeUnit.HOURS

7th test

"Omega"

Runnable.class

TimeUnit.DAYS

8th test

"Omega"

Runnable.class

TimeUnit.HOURS

9th test

"Omega"

Cloneable.class

TimeUnit.DAYS

10th test

"Omega"

Cloneable.class

TimeUnit.HOURS

11th test

"Omega"

Predicate.class

TimeUnit.DAYS

12th test

"Omega"

Predicate.class

TimeUnit.HOURS

Remember, you can reuse the same argument provider method, by explicitly passing its name to @CartesianProductTest’s `factory attribute.

@CartesianProductTest(factory = "provideArguments")
void testNeedingArguments(String string, int i) {
    // passing test code
}

@CartesianProductTest(factory = "provideArguments")
void testNeedingSameArguments(String string, int i) {
    // different passing test code
}

static CartesianProductTest.Sets provideArguments() {
    return new CartesianProductTest.Sets()
        .add("Mercury", "Earth", "Venus")
        .add(1, 12, 144);
}

Conditions for the static factory method

There are multiple conditions the static factory method has to fulfill to qualify:

  • must have the same name as the test method (or its name must be specified via the factory attribute)

  • must be static

  • must have no parameters

  • must return CartesianProductTest.Sets

  • must register values for every parameter exactly once

  • must register values in order

Returning wrong Sets in the static factory method

If you register too few, too many, or conflicting parameters, you will get an ParameterResolutionException. "Conflicting parameters" means your test method has a parameter that should be injected by JUnit (e.g.: TestReporter) but you also try to inject it.

Examples of badly configured tests/static factory method:

@CartesianProductTest(factory = "resolveParameters")
void tooFewParameters(String string, int i, boolean b) {
    // fails because the boolean parameter is not resolved
}

@CartesianProductTest(factory = "resolveParameters")
void tooManyParameters(String string) {
    // fails because we try to supply a non-existent integer parameter
}

@CartesianProductTest(factory = "resolveParameters")
void wrongOrderParameters(int i, String string) {
    // fails because the static factory method declared parameter sets in the wrong order
}

@CartesianProductTest(factory = "resolveTestReporterParam")
void conflictingParameters(String string, TestReporter info) {
    // fails because both the factory method and JUnit tries to inject TestReporter
}

static CartesianProductTest.Sets resolveParameters() {
    return new CartesianProductTest.Sets()
        .add("A", "B", "C")
        .add(1, 2, 3);
}

static CartesianProductTest.Sets resolveTestReporterParam() {
    return new CartesianProductTest.Sets()
        .add("A", "B", "C")
        .add(new MyTestReporter()); // in this case MyTestReporter implements TestReporter
}

Writing your own @ArgumentsSource for @CartesianProductTest

You might find that the available @ArgumentsSource annotations do not fit your need. In that case, you can write a custom @ArgumentsSource and a corresponding ArgumentsProvider for @CartesianProductTest.

Let’s demonstrate with an example.

For the sake of the example, let’s imagine that @CartesianValueSource does not exist. We would like to have an @ArgumentsSource, where we can specify integers. Let’s create an annotation for it.

import java.lang.annotation.*;

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Repeatable(CartesianIntSources.class)
@ArgumentsSource(IntArgumentsProvider.class)
public @interface CartesianIntSource {

    int[] numbers();

    @interface CartesianIntSources {
        CartesianIntSource[] value();
    }
}

The annotation has to have RUNTIME retention, so JUnit (and Pioneer) can discover it on your test via reflection. It does not have to be repeatable, necessarily, but it is strongly recommended, since you can only specify the input of a single test parameter in a single annotation. It has to be annotated with @ArgumentsSource, so Pioneer knows which ArgumentsProvider it should invoke.

Next, we need to have a class that takes these values and passes them to our test.

import java.util.Arrays;
import java.util.stream.Stream;

import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.ArgumentsProvider;

class IntArgumentsProvider implements ArgumentsProvider, CartesianAnnotationConsumer<CartesianIntSource> {

    private CartesianIntSource source;

    @Override
    void accept(CartesianIntSource source) {
        this.source = source;
    }

    @Override
    public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
        return Arrays.stream(source.value()).map(Arguments::of);
    }

}

The class has to implement both interfaces (ArgumentsProvider and CartesianAnnotationConsumer<? extends Annotation>). Note that an implementation of ArgumentsProvider must be declared as either a top-level class or as a static nested class. The IntArgumentsProvider class first 'consumes' the annotation via the accept method, then provides the arguments to JUnit via the provideArguments method. The order of these operations is guaranteed, so accept is always first and provideArguments is always second.

In our case, we don’t have to process the values we pass in the annotation, so we just return the values as is, but you could do additional processing, for example:

// no annotations to make example shorter
@interface PeopleSource {

    String[] names();

    int ages();

}

// no class definition to make example shorter
@Override
public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
    return IntStream.range(0, source.names().length)
        .mapToObj(i -> new Person(source.names()[i], source.ages()[i]))
        .map(Arguments::of);
}

Customizing Display Names

By default, the display name of a CartesianProductTest invocation contains the invocation index and the String representation of all arguments for that specific invocation. You can customize invocation display names via the name attribute of the @CartesianProductTest annotation. For example:

@CartesianProductTest(value = {"0", "1"}, name = "{index} => first bit: {0} second bit: {1}")
@DisplayName("Basic bit test")
void testWithCustomDisplayName(String a, String b) {
    // passing test code
}

When executing the above test, you should see output similar to the following:

Basic bit test
├─ 1 => first bit: 0 second bit: 0
├─ 2 => first bit: 0 second bit: 1
├─ 3 => first bit: 1 second bit: 0
└─ 4 => first bit: 1 second bit: 1

Please note that name is a MessageFormat pattern. A single quote (') needs to be represented as a doubled single quote ('') in order to be displayed.

CartesianProductTest supports the following placeholders in custom display names:

Placeholder Description

{displayName}

the display name of the method

{index}

the current invocation index, starting with 1

{arguments}

the complete, comma-separated arguments list

{0}, {1}, …​

an individual argument

Warning: Do not @CartesianProductTest with @Test

If @CartesianProductTest is combined with @Test or TestTemplate-based mechanisms (like @RepeatedTest or @ParameterizedTest), the test engine will execute it according to each annotation (i.e. more than once). This is most likely unwanted and will probably lead to the following exception/failure message:

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter […​]

This is because @Test does not know what to do with the parameter(s) of the @CartesianProductTest.

Thread-Safety

This extension is safe to use during parallel test execution.