JUnit: Excluding data driven tests

Once in a while there is that single test case that cannot be executed for certain data when running data driven tests. This could be a true  annoyment and it is tempting to break out those tests into a separate test class, don’t do that and stay away from taking shortcuts that will give you;

  • False test statistics, reporting passing test that has never been run
  • Avoid Ignoring tests e.g. using conditional ignores or JUnit’s assume.

A simple way to exclude a test from being executed when running data driven tests using the Parameterized.class is to set up a conditional JUnit rule. A test rule that based on the current test data in the data driven loop runs OR excludes a specifc test.

Consider the scenario below

We run the all tests below with the following samples “a”, “b”, “c” but know that test02 will not work for the sample “b” so we will have to deal with that in a clever way.

@Runwith(Parameterized.class)
public class Test {

  private String sample;

  public Test(String sample) {
    this.sample = sample;
  }

  @Parameters
  public static Collection<Object[]> generateSamples() {
    final List <Object[]> samples = new ArrayList<Object[]>();
    sample.add(new Object[]{"a"});
    sample.add(new Object[]{"b"});
    sample.add(new Object[]{"c"});
    return samples;
  }

  @Test
  public void test01() {
    // Works with sample "a", "b", "c"
  }

  @Test
  public void test02() {
    // Works with sample "a", "c" BUT NOT "b"
  }

  @Test
  public void test03() {
    // Works with sample "a", "b", "c"
  }
}

Mark-up with test annotation

Step one is to mark-up a test case so that we know that it will only run for certain samples. We will do this by adding an annotation, in this case we call it Samples.

@Test
public void test01() { ... }

@Samples({"a","c"})
@Test
public void test02() { ... }

@Samples({"a","b","c"})
@Test
public void test03() { ... }

Using the test annotation Samples we can start filtering during run-time if to execute a test case or not. A test without the annotation will always run the test no matter what data that is reurned in the data driven loop. If the annotation is in place the test will only run for the samples that matches the values in the annotation. In case of test02 it will only run for samples “a” and “c”.

The test annotation

Adding the annotation is straight forward, read up on it here.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Samples {
    String [] value();
}

Adding the exclusion rule

Worth knowing is that JUnit rules will always be created before a test class constructor. This means that a rule will not know the test data sample of the current data driven iteration. The current data driven value is always passed to the test class constructor (Parameterized.class) and hence the rule we are going to implement needs to get that information as well at that point.

@Runwith(Parameterized.class)
public class Test {

  @Rule
  public OnlyRunForSampleRule rule = new OnlyRunForSampleRule();
  private String sample;

  public Test(String sample) {
    this.sample = sample;
    rule.setSample(sample); // <<-- HERE
  }

With the current data sample and the Samples annotation values known at run-time, we define a simple rule that excludes test cases if they are not supposed to run for certain test data.

public class OnlyRunForSampleRule implements TestRule {

  private String sample;

  @Override
  public Statement apply(Statement s, Description d) {
    Protocols annotation = d.getAnnotation(Samples.class);
    // No annotation/samples matching, always run
    if (annotation == null) {
      return s;
    }
    // Match! One sample value matches current parameterized sample value
    else if (Arrays.asList(annotation.value()).contains(protocol)) {
      return s;
    }
    // No match in the samples annotation, skipping
    return new Statement() {
      @Override
      public void evaluate() throws Throwable {}
    };
  }

  public void setSample(String sample) {
    this.sample = sample;
  }
}

The rule above is defined to run tests on known samples, creating the inverse of this rule is simply done by returning the empty Statement in the else if that checks for matches in the Samples annotation.