From Wikipedia:

In mathematics, specifically set theory, the Cartesian product of two sets A and B, denoted A × B, is the set of all ordered pairs (a, b) where a is in A and b is in B. In terms of set-builder notation, that is A × B = {(a,b) | a ∈ A and b ∈ B} [...] One can similarly define the Cartesian product of n sets, also known as an n-fold Cartesian product, which can be represented by an n-dimensional array, where each element is an n-tuple.

What does all this mean?

The Cartesian product of sets is all the possible combinations where you take a single element from each set. If you have two sets, { 1, 2 } and { 3, 4 }, their Cartesian product is { { 1, 3 }, { 1, 4 }, { 2, 3 }, { 2, 4 } }.

Sometimes it’s useful to test all possible combinations of parameter sets. Normally, this results in a lot of written test data parameters. For a more comfortable way you may use the @CartesianTest extension. The extension takes the test data parameter values and runs the test for every possible combination of them.

Note

The CartesianTest extension has undergone significant changes in 1.6.0. This here is the new variant, which, among other differences, lives in another package and does much of its work by annotating parameters directly. The old variant was deprecated in 1.6.0 and removed in 2.0.

Basic use

@CartesianTest is used instead of @Test or other such annotations (e.g. @RepeatedTest).

You can supply test parameters to @CartesianTest in two ways:

  • You can annotate the test parameters directly with @CartesianTest.Values, @CartesianTest.Enum, or range source annotations (see Defining arguments on parameters)

  • You can annotate test method itself with @CartesianTest.MethodFactory to point at a factory method that creates sets of arguments (see Defining arguments with factories)

Specifying more than one kind of parameter source (i.e.: annotating your test parameters and the test method itself) does not work and will throw an ExtensionConfigurationException.

Our earlier example with { 1, 2 } and { 3, 4 }, would look like this:

@CartesianTest
void myCartesianTestMethod(
        @Values(ints = { 1, 2 }) int x,
        @Values(ints = { 3, 4 }) int y) {
    // passing test code
}

@CartesianTest works with parameters injected by JUnit automatically (e.g.: TestReporter). You can read about auto-injected parameters here.

Just like the mathematical Cartesian product, @CartesianTest works with sets. Duplicate elements get removed automatically. If your input is { 1, 1, 3 } and { 2, 2 } the extension will consider their Cartesian product { { 1, 2 }, { 3, 2 } }. Otherwise, the test would run with the same parameters multiple times. If you need to pass the same parameters multiple times, you might want to look into repeated tests.

Defining arguments on parameters

You can annotate the parameters of your @CartesianTest, to provide values to that specific parameter. Parameter annotations are "self-contained", they only provide values to the parameter they are on and do not interfere with each other. You can mix and match parameter annotations as you need.

@CartesianTest
void testIntChars(
        @ShortRangeSource(from = 1, to = 3, step = 1) short s,
        @Values(strings = { "A", "B" }) String character,
        @Enum ChronoUnit unit) {
    // passing test code
}

@CartesianTest.Values

@CartesianTest.Values is used to define the possible inputs of a single test parameter. The test will try every combination those values can have.

@CartesianTest
void testIntChars(
        @Values(ints = { 1, 2, 4 }) int number,
        @Values(strings = { "A", "B" }) String character) {
    // passing test code
}

This annotation might look familiar - it mimics JUnit’s @ValueSource, except @CartesianTest.Values must be put on the parameter. It does NOT work with @ParameterizedTest.

The test testIntChars is executed exactly six times. The first parameter can have any of the three values 1, 2 or 4. The second parameter can have any of the two values "A" or "B". @CartesianTest tests for all input combinations, that’s 3 × 2, so six tests in total.

To demonstrate with a table:

# of test value of number value of character

1st test

1

"A"

2nd test

1

"B"

3rd test

2

"A"

4th test

2

"B"

5th test

4

"A"

6th test

4

"B"

@CartesianTest.Enum

@CartesianTest.Enum provides a convenient way to use Enum constants.

@CartesianTest
void testWithEnum(@Enum ChronoUnit unit) {
    assertThat(unit).isNotNull();
}

Like @CartesianTest.Values, @CartesianTest.Enum is an annotation that might look familiar - it mimics JUnit’s @EnumSource, except @CartesianTest.Enum must be put on the parameter. It does NOT work with @ParameterizedTest.

The annotation has an optional value attribute. When omitted, the declared type of the parameter is used. The test will fail if it is not an enum type. The value attribute is required in the following example because the method parameter is declared as TemporalUnit, i.e. the interface implemented by ChronoUnit, which isn’t an enum type.

@CartesianTest
void testExplicitEnum(
        @Enum(ChronoUnit.class) TemporalUnit unit) {
    assertThat(unit).isNotNull();
}

The annotation provides an optional names attribute that lets you specify which constants shall be used, like in the following example. If omitted, all constants will be used.

@CartesianTest
void testEnumNames(
        @Enum(names = { "DAYS", "HOURS" }) ChronoUnit unit) {
    assertThat(EnumSet.of(ChronoUnit.DAYS, ChronoUnit.HOURS)).contains(unit);
}

The annotation also provides an optional mode attribute that enables fine-grained control over which constants are passed to the test method. For example, you can exclude names from the enum constant pool or specify regular expressions as in the following examples.

@CartesianTest
void testWithEnumModes(
        @Enum(mode = EXCLUDE, names = { "ERAS", "FOREVER" })
        ChronoUnit unit) {
    assertThat(EnumSet.of(ChronoUnit.ERAS, ChronoUnit.FOREVER)).doesNotContain(unit);
}
@CartesianTest
void testWithEnumRegex(
        @Enum(mode = MATCH_ALL, names = "^.*DAYS$") ChronoUnit unit) {
    assertThat(unit.name()).endsWith("DAYS");
}

The example below shows how to use @CartesianTest.Enum with two Enum types.

enum MyEnum {
    ONE, TWO, THREE
}

enum AnotherEnum {
    ALPHA, BETA, GAMMA, DELTA
}

@CartesianTest
void testEnumValues(
        @Enum MyEnum myEnum,
        @Enum(names = { "ALPHA", "DELTA" }, mode = Enum.Mode.EXCLUDE)
        AnotherEnum anotherEnum) {
    // passing test code
}

The test testEnumValues is executed exactly six times. The first parameter can have any of the three constants ONE, TWO or THREE. The second parameter can have any of the two constants BETA or GAMMA (note the EXCLUDE mode applied to the other two constants). @CartesianTest tests for all input combinations, that’s 3 × 2, so six tests in total.

To demonstrate with a table:

# of test value of myEnum value of anotherEnum

1st test

ONE

BETA

2nd test

ONE

GAMMA

3rd test

TWO

BETA

4th test

TWO

GAMMA

5th test

THREE

BETA

6th test

THREE

GAMMA

Range source annotations

You can annotate your test parameters with range source annotations. For this purpose only, range sources can be used on parameters.

@CartesianTest
void testShortAndLong(
        @ShortRangeSource(from = 1, to = 3, step = 1) short s,
        @LongRangeSource(from = 0L, to = 2L, step = 1, closed = true) long l) {
    // passing test code
}

The test testShortAndLong is executed exactly six times. The first parameter can have any of the two values 1, 2. The second parameter can have any of the three values 0L, 1L or 2L. @CartesianTest tests for all input combinations, that’s 2 × 3, so six tests in total.

To demonstrate with a table:

# of test value of s value of l

1st test

1

0L

2nd test

1

1L

3rd test

1

2L

4th test

2

0L

5th test

2

1L

6th test

2

2L

For more information, please see the separate documentation about range sources.

Defining arguments with factories

You can annotate your test method to supply arguments to all parameters simultaneously.

@CartesianTest.MethodFactory

@CartesianTest.MethodFactory can be used to name a factory method that supplies your arguments. The value annotation parameter is mandatory. Just like with JUnit’s @MethodSource, you can specify the factory method with its fully-qualified name (including the class), e.g. com.example.Class#factory. This method must return ArgumentSets.

ArgumentSets is a helper class, specifically for creating sets for @CartesianTest. To create the test data, instantiate with the static factory method argumentsForFirstParameter, then call the addValuesForNextParameter method once per additional parameter in the order in which they appear in the test method. In each call, pass in all values for the corresponding parameter. For convenience, all methods return with your ArgumentSets instance, so you can chain add…​ calls. If you want to create an initially-empty ArgumentSets, call the static factory method create().

Let’s look at an example.

@CartesianTest
@CartesianTest.MethodFactory("stringClassTimeUnitFactory")
void testStringClassTimeUnit(
        String string,
        Class<?> clazz,
        TimeUnit unit) {
    // passing test code
}

static ArgumentSets stringClassTimeUnitFactory() {
    return ArgumentSets
            .argumentsForFirstParameter(
                    "Alpha", "Omega")
            .argumentsForNextParameter(
                    Runnable.class, Cloneable.class, Predicate.class)
            .argumentsForNextParameter(
                    TimeUnit.DAYS, TimeUnit.HOURS);
}

The test testMethod is executed exactly twelve times. The first parameter can have any of the two values "Alpha" or "Omega". The second parameter can have any of the three values Runnable.class, Cloneable.class or Predicate.class. The third parameter can have any of the two values TimeUnit.DAYS or TimeUnit.HOURS. @CartesianTest tests for all input combinations, that’s 2 × 3 × 2, so twelve tests in total.

To demonstrate with a table:

# of test value of string value of clazz value of unit

1st test

"Alpha"

Runnable.class

TimeUnit.DAYS

2nd test

"Alpha"

Runnable.class

TimeUnit.HOURS

3rd test

"Alpha"

Cloneable.class

TimeUnit.DAYS

4th test

"Alpha"

Cloneable.class

TimeUnit.HOURS

5th test

"Alpha"

Predicate.class

TimeUnit.DAYS

6th test

"Alpha"

Predicate.class

TimeUnit.HOURS

7th test

"Omega"

Runnable.class

TimeUnit.DAYS

8th test

"Omega"

Runnable.class

TimeUnit.HOURS

9th test

"Omega"

Cloneable.class

TimeUnit.DAYS

10th test

"Omega"

Cloneable.class

TimeUnit.HOURS

11th test

"Omega"

Predicate.class

TimeUnit.DAYS

12th test

"Omega"

Predicate.class

TimeUnit.HOURS

You can reuse the same argument provider method multiple times.

@CartesianTest
@CartesianTest.MethodFactory("stringIntFactory")
void testStringInt(String string, int i) {
    // passing test code
}

@CartesianTest
@CartesianTest.MethodFactory("stringIntFactory")
void testSameStringInt(String string, int i) {
    // different passing test code
}

static ArgumentSets stringIntFactory() {
    return ArgumentSets
            .argumentsForFirstParameter("Mercury", "Earth", "Venus")
            .argumentsForNextParameter(1, 12, 144);
}

You can make the argument provider method non-static if the test class is annotated with @TestInstance(Lifecycle.PER_CLASS).

@Nested
// with this lifecycle annotation the factory method can be non-static
@TestInstance(Lifecycle.PER_CLASS)
class MyCartesianTests {

    @CartesianTest
    @CartesianTest.MethodFactory("provideArguments")
    void testNeedingArguments(String string, int i) {
        // passing test code
    }

    // this provider method doesn't have to be static
    ArgumentSets provideArguments() {
        return ArgumentSets
                .argumentsForFirstParameter(
                        "Mercury", "Earth", "Venus")
                .argumentsForNextParameter(1, 12, 144);
    }

}

Requirements for the factory method

There are multiple requirements the factory method has to fulfill to qualify:

  • must have the same name as the test method (or its name must be specified via the factory attribute)

  • must be static (unless the test class is annotated with @TestInstance(Lifecycle.PER_CLASS) and the factory method is defined in the test class)

  • must have no parameters

  • must return ArgumentSets

  • must register values for every parameter exactly once

  • must register values in order

Returning wrong ArgumentSets in the factory method

If you register too few, too many, or conflicting parameters, you will get an ParameterResolutionException. "Conflicting parameters" means your test method has a parameter that should be injected by JUnit (e.g.: TestReporter) but you also try to inject it.

Examples of badly configured tests/factory method:

@CartesianTest
@CartesianTest.MethodFactory("unsuitableStringIntFactory")
void testHasTooFewParameters(String string) {
    // fails because we try to supply a non-existent integer parameter
}

@CartesianTest
@CartesianTest.MethodFactory("unsuitableStringIntFactory")
void testHasTooManyParameters(String string, int i, boolean b) {
    // fails because the boolean parameter is not resolved
}

@CartesianTest
@CartesianTest.MethodFactory("unsuitableStringIntFactory")
void testHasParametersInWrongOrder(int i, String string) {
    // fails because the factory method declared
    // parameter sets in the wrong order
}

@CartesianTest
@CartesianTest.MethodFactory("resolveTestReporterParam")
void testHasConflictingParameters(String string, TestReporter info) {
    // fails because both the factory method and JUnit
    // try to inject TestReporter
}

static ArgumentSets unsuitableStringIntFactory() {
    return ArgumentSets
            .argumentsForFirstParameter("A", "B", "C")
            .argumentsForNextParameter(1, 2, 3);
}

static ArgumentSets resolveTestReporterParam() {
    return ArgumentSets
            .argumentsForFirstParameter("A", "B", "C")
            // in this case MyTestReporter implements TestReporter
            .argumentsForNextParameter(new MyTestReporter());
}

Writing your own @ArgumentsSource for @CartesianTest

You might find that the available annotations do not fit your need. For a @ParameterizedTest, you can write a custom @ArgumentsSource and a corresponding ArgumentsProvider. You can do the same thing for @CartesianTest, with the following caveats:

  • Instead of @ArgumentsSource, you have to use @CartesianArgumentsSource

  • Instead of ArgumentsProvider, you have to use one of two interfaces, each explored in their own section.

Implementing CartesianParameterArgumentsProvider

Let’s see how you can provide arguments to a single parameter with a custom annotation and provider. You will have to use CartesianParameterArgumentsProvider instead of ArgumentsProvider. This will let you initialize your argument source with its corresponding parameter.

Let’s demonstrate with an example.

For the sake of simplicity, let’s imagine that @CartesianTest.Values does not exist. We would like to have an argument source, where we can specify integers. Let’s create an annotation for it.

@Target(ElementType.PARAMETER)
@Retention(RetentionPolicy.RUNTIME)
@CartesianArgumentsSource(IntArgumentsProvider.class)
public @interface Ints {

    int[] value();

}

The annotation targets parameters because we want to use it directly on a parameter. The annotation has RUNTIME retention, so JUnit (and Pioneer) can discover it on your test via reflection. It is annotated with @CartesianArgumentsSource, so Pioneer knows which CartesianParameterArgumentsProvider it should invoke.

Next, we need to implement IntArgumentsProvider, that takes these values and passes them to our test.

class IntArgumentsProvider
        implements CartesianParameterArgumentsProvider<Integer> {

    @Override
    public Stream<Integer> provideArguments(
            ExtensionContext context, Parameter parameter) {
        Ints source = Objects.requireNonNull(
                parameter.getAnnotation(Ints.class));
        return Arrays.stream(source.value()).boxed();
    }

}

The class has to implement CartesianParameterArgumentsProvider. Note that an implementation of CartesianParameterArgumentsProvider must be declared as either a top-level class or as a static nested class. It must also provide a default (no argument) constructor. The second parameter of the provideArguments method is the Parameter object that represents the parameter for which you are supplying arguments.

In our case, we don’t have to process the values from our annotation, so we just return the values as is. You could do additional processing, for example:

@Target(ElementType.PARAMETER)
@Retention(RetentionPolicy.RUNTIME)
@CartesianArgumentsSource(PeopleProvider.class)
@interface People {

    String[] names();

    int[] ages();

}

class PeopleProvider implements CartesianParameterArgumentsProvider<Person> {

    @Override
    public Stream<Person> provideArguments(ExtensionContext context, Parameter parameter) {
        People source = Objects.requireNonNull(parameter.getAnnotation(People.class));
        return IntStream
                .range(0, source.names().length)
                .mapToObj(i -> new Person(source.names()[i], source.ages()[i]));
    }

}

Alternatively, @CartesianTest supports it if you want to implement the AnnotationConsumer interface. The previous example would look like the following:

class PeopleProviderWithAnnotationConsumer
        implements CartesianParameterArgumentsProvider<Person>, AnnotationConsumer<People> {

    private People source;

    @Override
    public Stream<Person> provideArguments(ExtensionContext context, Parameter parameter) {
        return IntStream
                .range(0, source.names().length)
                .mapToObj(i -> new Person(source.names()[i], source.ages()[i]));
    }

    @Override
    public void accept(People source) {
        this.source = source;
    }

}

Implementing CartesianMethodArgumentsProvider

Let’s see how you can provide arguments to every parameter with a custom annotation and provider. You will have to use CartesianMethodArgumentsProvider instead of ArgumentsProvider.

Let’s demonstrate with an example.

We would like to have an argument that supplies 'bits' to every parameter. Let’s create an annotation for it.

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@CartesianArgumentsSource(BitArgumentsProvider.class)
public @interface BitSource {
}

The annotation targets methods because we want to use it on the test method itself. The annotation has RUNTIME retention, so JUnit (and Pioneer) can discover it on your test via reflection. It is annotated with @CartesianArgumentsSource, so Pioneer knows which CartesianMethodArgumentsProvider it should invoke.

Next, we need to implement BitArgumentsProvider.

class BitArgumentsProvider implements CartesianMethodArgumentsProvider {

    @Override
    public ArgumentSets provideArguments(ExtensionContext context) {
        int paramCount = context.getRequiredTestMethod().getParameters().length;
        ArgumentSets sets = ArgumentSets.create();
        for (int i = 0; i < paramCount; i++) {
            sets.argumentsForNextParameter(0, 1);
        }
        return sets;
    }

}

The class has to implement CartesianMethodArgumentsProvider. Note that an implementation of CartesianMethodArgumentsProvider must be declared as either a top-level class or as a static nested class. It must also provide a default (no argument) constructor. The method provideArguments returns an ArgumentSets object. This object should contain the arguments list for every parameter, except if an argument is supplied by JUnit (e.g.: a TestInfo).

Warning
 This example does not work with parameters that are supplied by JUnit, you have to add custom logic for that scenario.

@CartesianTest supports it if you want to implement the AnnotationConsumer interface. For example:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@CartesianArgumentsSource(NumberArgumentsProvider.class)
public @interface NumberSource {

    int[] value();

}

static class NumberArgumentsProvider implements CartesianMethodArgumentsProvider, AnnotationConsumer<NumberSource> {

    private int[] numbers;

    @Override
    public ArgumentSets provideArguments(ExtensionContext context) {
        int paramCount = context.getRequiredTestMethod().getParameters().length;
        ArgumentSets sets = ArgumentSets.create();
        for (int i = 0; i < paramCount; i++) {
            sets.argumentsForNextParameter(Arrays.stream(numbers).boxed());
        }
        return sets;
    }

    @Override
    public void accept(NumberSource source) {
        this.numbers = source.value();
    }

}

Customizing Display Names

By default, the display name of a CartesianTest invocation contains the invocation index and the String representation of all arguments for that specific invocation. You can customize invocation display names via the name attribute of the @CartesianTest annotation. For example:

@CartesianTest(name = "{index} => first bit: {0} second bit: {1}")
@DisplayName("Basic bit test")
void testWithCustomDisplayName(
        @Values(strings = { "0", "1" }) String a,
        @Values(strings = { "0", "1" }) String b) {
    // passing test code
}

When executing the above test, you should see output similar to the following:

Basic bit test
├─ 1 => first bit: 0 second bit: 0
├─ 2 => first bit: 0 second bit: 1
├─ 3 => first bit: 1 second bit: 0
└─ 4 => first bit: 1 second bit: 1

Please note that name is a MessageFormat pattern. A single quote (') needs to be represented as a doubled single quote ('') in order to be displayed.

CartesianTest supports the following placeholders in custom display names:

Placeholder Description

{displayName}

the display name of the method

{index}

the current invocation index, starting with 1

{arguments}

the complete, comma-separated arguments list

{0}, {1}, …​

an individual argument

Do not use @CartesianTest with @Test

If @CartesianTest is combined with @Test or TestTemplate-based mechanisms (like @RepeatedTest or @ParameterizedTest), the test engine will execute it according to each annotation (i.e. more than once). This is most likely unwanted and will probably lead to the following exception/failure message:

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter […​]

This is because @Test does not know what to do with the parameter(s) of the @CartesianTest.

Thread-Safety

This extension is safe to use during parallel test execution.