Disable Tests if Others Fail

There are situations where, when one test in a class fails, the remaining tests in the class don’t need to be executed because their results offer little additional insight. In those situations, disabling the remaining tests can be preferable in order to avoid larger build logs and/or reduce the test suite’s overall run time. By applying @DisableIfTestFails to the test class, all tests after the first failed one are disabled.

@DisableIfTestFails

When applied to a type, @DisableIfTestFails will disable all tests that are launched after the first failing test in that class:

@DisableIfTestFails
// this annotation ensures that tests are run in the order of
// the numbers passed to the `@Order` annotation below
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class Tests {

    @Test
    @Order(1)
    void test1() {
    }

    @Test
    @Order(2)
    void test2() {
        fail();
    }

    @Test
    @Order(3)
    void test3() {
    }

}

In this example, test1 will pass and test2 will fail with or without the extension but due to @DisableIfTestFails, test3 will be disabled (instead of being executed).

The extension ignores aborted tests (e.g. due to failed assumptions) and won’t disable tests because of that. By default, it will disable remaining tests on any exception, including failed assertions - both of that can be configured.

Not disabling on failed assertions

By default, @DisableIfTestFails will disable remaining tests on failed assertions. If instead, failed assertions are acceptable and only other test failures should lead to disabling, set onAssertion = false:

@DisableIfTestFails(onAssertion = false)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class ThreeTestsWithSecondFailingWithUnconfiguredAssertionTestCase {

    @Test
    @Order(1)
    void test1() {
    }

    @Test
    @Order(2)
    void test2() {
        // fails test with assertion
        assertTrue(false);
    }

    @Test
    @Order(3)
    void test3() {
    }

}

The test test2 fails because of an assertion but due to onAssertion = false, test3 isn’t disabled and will execute.

Disabling on specific exceptions

By default, @DisableIfTestFails will disable remaining tests on any exception. If instead, only specific exceptions should lead to disabling, define them with with:

@DisableIfTestFails(with = IOException.class)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class ThreeTestsWithSecondThrowingConfiguredExceptionTestCase {

    @Test
    @Order(1)
    void test1() {
    }

    @Test
    @Order(2)
    void test2() throws InterruptedException {
        throw new InterruptedException();
    }

    @Test
    @Order(3)
    void test3() throws IOException {
        throw new IOException();
    }

    @Test
    @Order(4)
    void test4() {
    }

}

The test test2 fails, but because the with attribute doesn’t include its exception’s type InterruptedException, tests will continue to be executed. Then test3 fails with the configured exception and so test4 is disabled.

Inheritance

The annotation can be applied to classes and interfaces that have further implementations, which means any specific test class might inherit/implement from multiple types that have @DisableIfTestFails annotations. In that case, the extension disables the maximum number of tests:

• the exception types given to with are merged, i.e. tests are disabled if a test fails with any of the configured exceptions

• onAssertion is or’ed, i.e. only if all are configured to keep executing on a failed assertion, will that happen

But if a test fails in a specific class, only other tests in the same class will be disabled. That means if…​

• a test class SpecificTests implements interface Tests and

• Tests is annotated with @DisableIfTestFails and

• a test in SpecificTests fails

  1. then, only remaining tests in SpecificTests are disabled and other implementations of Tests remain unaffected, i.e. their tests are not disabled.

Thread-Safety

This extension is safe to use during parallel test execution, but no guarantee can be given how soon after a failed test other tests will be disabled, i.e. how soon other threads observe that "their" tests are supposed to be disabled.