Using JUnit @Category and Maven profiles to get stable test suites

Since it happens over and over again, no matter what project I am in. Keeping the test suites green and passing is a …

The blind eye

The one sneaking little test case that starts failing can prove to be a real killer to all automation approaches. More than once it has shown that as soon as a single test case breaks a test suite things starts to go downhill, fast. Ok, there could be acceptable resons for the test case to be broken, e.g. if there are issues in the system under test or in the test framework that are not bugs.

The usual decision taken is to leave the test case as is for now since we are all aware that it will fail for a while, BAD choice!!! All attention on the failing test suite is moving from slight attention to abandoned, and if other test in the test same test suite starts failing for various reasons it will most likely go by undetected.

What can/should be done here

• Disable the test case and enable it again when it should be back on track
• Maintain test suites to allow it to execute and fail but in a another suite/run to keep the MUST always pass test suites all green

The later case is the way to go.

Maven profiles to your rescue

Using Maven profiles it is easy to group your JUnit test classes and test cases in different categories. Usually this is used for grouping tests into slow, fast and browser-less tests a.s.f.. This approach can be stretched even further by using a grouping that reflects the current state of the test case. There does not have to be that many different groupable categories but at least two is needed.

• Stable
• Unstable/Maturing

Stable is self explained while Unstable/Maturing covers everything that is NOT always PASSING. It might be test cases that are unstable beacuse of;

• timing issues
• tested functionality is still under development (during sprints)
• there is a low priority bug that will be fixed later
• unknown issues causing it to sporadically fail
• waiting to get a STABLE stamp

Use case 1
The last item is an approach that can be used to be really be sure that the stable test suites does not get polluted. Any new test case needs to run a set number of times (e.g. 20) in a test suite and must pass all all times before being stamped as a reliable test case.

Use case 2
When a bug surfaces that causes a test case to fail it might sometimes be the case that it wont be fixed for some time period. Instead of totally disabling or trying to fix the test case that would fail an otherwise stable test suite it shall be moved to an unstable test suite. This is a way to put the test case in a quarantine until the bug has been fixed and on the same tame make sure it still compiles and is executable.

Categorizing tests using JUnit @Category annotation
Use the JUnit @Category annotation in front of your test cases or test classes.

@Category(x.y.z.Stable.class)
@Test
public void aTestCase() {
  // void
}

// 
// Alternatively an unstable test case
//

@Category(x.y.z.Unstable.class)
@Test
public void anotherTestCase() {
  // void
}

Splitting test execution using Maven profiles

Configure your Surefire plugin to include and exclude certain test classes as well as determine a set of JUnit @Category groups to be included in the test run. Set up your pom including profiles as examplified below.

<build>	
	<plugins>	
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>2.12.2</version>
			<dependencies>
				<dependency>
					<groupId>org.apache.maven.surefire</groupId>
					<artifactId>surefire-junit47</artifactId>
					<version>2.12.2</version>
				</dependency>
			</dependencies>
			<configuration>
				<testFailureIgnore>true</testFailureIgnore>
				<excludes>
					<exclude>${exclude.tests}</exclude>
				</excludes>
				<includes>
					<include>${include.tests}</include>
				</includes>
				<groups>${testcase.groups}</groups>
				<configuration>
					<forkMode>never</forkMode>
					<runOrder>random</runOrder>
				</configuration>
			</configuration>
		</plugin>
	</plugins>
</build>

The testcase.groups property can be used to define a set of groups (JUnit @Categories) to include when tests using the actual profile.

<profiles>
	<profile>
		<id>stable-tests</id>
		<properties>
			<exclude.tests>**/x/**/*.java</exclude.tests>
			<include.tests>**/y/**/*.java</include.tests>
			<testcase.groups>x.y.z.Stable</testcase.groups>
		</properties>
	</profile>
	<profile>
		<id>unstable-tests</id>
		<properties>
			<exclude.tests>**/x/**/*.java</exclude.tests>
			<include.tests>**/y/**/*.java</include.tests>
			<testcase.groups>x.y.z.Unstable,x.y.z.Maturing</testcase.groups>
		</properties>
	</profile>
</profiles>