JUnit: Excluding data driven tests

Once in a while there is that single test case that cannot be executed for certain data when running data driven tests. This could be a true  annoyment and it is tempting to break out those tests into a separate test class, don’t do that and stay away from taking shortcuts that will give you;

  • False test statistics, reporting passing test that has never been run
  • Avoid Ignoring tests e.g. using conditional ignores or JUnit’s assume.

A simple way to exclude a test from being executed when running data driven tests using the Parameterized.class is to set up a conditional JUnit rule. A test rule that based on the current test data in the data driven loop runs OR excludes a specifc test.

Consider the scenario below

We run the all tests below with the following samples “a”, “b”, “c” but know that test02 will not work for the sample “b” so we will have to deal with that in a clever way.

public class Test {

  private String sample;

  public Test(String sample) {
    this.sample = sample;

  public static Collection<Object[]> generateSamples() {
    final List <Object[]> samples = new ArrayList<Object[]>();
    sample.add(new Object[]{"a"});
    sample.add(new Object[]{"b"});
    sample.add(new Object[]{"c"});
    return samples;

  public void test01() {
    // Works with sample "a", "b", "c"

  public void test02() {
    // Works with sample "a", "c" BUT NOT "b"

  public void test03() {
    // Works with sample "a", "b", "c"

Mark-up with test annotation

Step one is to mark-up a test case so that we know that it will only run for certain samples. We will do this by adding an annotation, in this case we call it Samples.

public void test01() { ... }

public void test02() { ... }

public void test03() { ... }

Using the test annotation Samples we can start filtering during run-time if to execute a test case or not. A test without the annotation will always run the test no matter what data that is reurned in the data driven loop. If the annotation is in place the test will only run for the samples that matches the values in the annotation. In case of test02 it will only run for samples “a” and “c”.

The test annotation

Adding the annotation is straight forward, read up on it here.

public @interface Samples {
    String [] value();

Adding the exclusion rule

Worth knowing is that JUnit rules will always be created before a test class constructor. This means that a rule will not know the test data sample of the current data driven iteration. The current data driven value is always passed to the test class constructor (Parameterized.class) and hence the rule we are going to implement needs to get that information as well at that point.

public class Test {

  public OnlyRunForSampleRule rule = new OnlyRunForSampleRule();
  private String sample;

  public Test(String sample) {
    this.sample = sample;
    rule.setSample(sample); // <<-- HERE

With the current data sample and the Samples annotation values known at run-time, we define a simple rule that excludes test cases if they are not supposed to run for certain test data.

public class OnlyRunForSampleRule implements TestRule {

  private String sample;

  public Statement apply(Statement s, Description d) {
    Protocols annotation = d.getAnnotation(Samples.class);
    // No annotation/samples matching, always run
    if (annotation == null) {
      return s;
    // Match! One sample value matches current parameterized sample value
    else if (Arrays.asList(annotation.value()).contains(protocol)) {
      return s;
    // No match in the samples annotation, skipping
    return new Statement() {
      public void evaluate() throws Throwable {}

  public void setSample(String sample) {
    this.sample = sample;

The rule above is defined to run tests on known samples, creating the inverse of this rule is simply done by returning the empty Statement in the else if that checks for matches in the Samples annotation.

Debugging WireMock calls when using JUnit WireMockRule

Mocking using the WireMockRule in your JUnit test classes and struggle with 404’s?

It is not that trivial to find in the WireMock documentation but it is in there, under ‘Listening for requests’ @ http://wiremock.org/verifying.html. Plain debugging fine, but sometimes one really wants to know the details of the calls made to the underlying services that are consumed, especially when WireMocking these services and there is a fine grained matching mechanism to deal with.

Below is the quick awesome tip to get the details you need to resolve the WireMock returned 404’s easily.

Add a request listener to your WireMockRule and Use Java 8 lambdas to smoothly implement the WireMock interface RequestListener that has the single method requestReceived(Request request, Response response). Print out the reponse and request details you want. Run your tests and check the print outs, all set!

import com.github.tomakehurst.wiremock.junit.WireMockRule;

public class Test {

   public WireMockRule wireMockRule = new WireMockRule(6969);

   public void setupTest() {
      wireMockRule.addMockServiceRequestListener((request, response) -> {
         System.out.println("URL Requested => " + request.getAbsoluteUrl());
         System.out.println("Request Body => " + request.getBodyAsString());
         System.out.println("Request Headers => " + request.getAllHeaderKeys());
         System.out.println("Response Status => " + response.getStatus());
         System.out.println("Response Body => " + response.getBodyAsString());

Overview of sporadically failing test cases in Jenkins – UnitTH Jenkins plugin

Never, ever a stable blue build right? It does not seem to matter what you do, but for system integration tests there always seems to one or a few sporadically failing test cases. What can be extra annoying is that there might always be different test cases that sporadically fail, not the same one or two. To be able to determine the success of executed test suites there is a need to get an overview of the last few runs. The easiest way to visualise this is to create a matrix of test runs versus test cases to get a hit map with frequency of failures for specific test cases.


Test case failure spread.

For those using Jenkins there is a small Jenkins plugin at Sourceforge that can be added to the post build steps of any job that generates test results. The plugin gives you useful overview and stats for the available build test results.



  • Edit the configuration of a Jenkins job and in the post build section select the plugin
  • Run the job again to get the first matrix report, there will be a link in the sidebar
  • Every failed run of a test case has a link to the report trace in the build it was executed, simply click on the red hits in the matrix

Test automation don’ts #1 – Separate repositories

Since this is one of these days when frustration is at what feels like an all time high ‘again’ what better to do that to get it all out of you. When it comes to test automation there are so many poor decisions out there… yes, many. I’ll try to address some of these here. The first one out is, da-da.

#1 Do not keep automation code in a separate reposity

So what does it mean, consider the case where all test cases and test framework code is in a nicely well structured repository and the code to be tested resides in an equally professionally handled repository. But what happens when it is time to run test cases? Is the HEAD of these repositories/branches ‘always’ in synch, well most likely they are not, not even if the codebase is completely ‘branch free’. The end result, most likely broken tests that will cause a lot of waste.

So for the case where there is one repository for the product code, DO NOT put the automation code in a sperate repository or you’ll end up struggling with failing test cases as soon as the branches get out of synch. On top of this guess what kind of overhead you will face if a team/teams are working with a feature branch strategy, let me smile and mention that this is what I struggle with today broken tests all over the place and no one has a clue about which failing tests that are regressions and which ones are expected to fail?

I have seen the scenario with separate repositories at several places and the usual reason is that the system (under test) it self is spread out over multiple repositories and the aim was to keep the automated tests in one place supporting all parts of a ‘system’. Not even tests that are defined as end-2-end tests spanning a full system will handle the scenario smoothly. I have been involved in several approaches trying to resolve problems without putting the automation code in the same repository, none of the approaches have been fully successful.

Identified problems

  • Test code out of synch with code under test
  • TWICE the amount of branches to maintain and work with

Bundling ChromeDriver with your test code

This post will just exemplify one way of maintaining the chrome drivers used for running automated tests with WebDriver and the Chrome browser without having to update and install the ChromeDriver to all possible nodes where tests will be running. The ChromeDriver will simply be bundled with the running tests and put under source control as for all other test ware.

Downloading links

Start by putting the the drivers under the resource folder so that they will be picked up by Maven per default.

<project root>/src/main/resources/chromedriver/mac/chromedriver
<project root>/src/main/resources/chromedriver/windows/chromedriver.exe

There are of course different drivers for different OS types and this needs to be handled using the os.name system property.

As per the ChromeDriver usage instructions (here) a system property has to be set pointing to the ChromeDriver server to use for the Chrome browser bridging. We will not point to a fixed location in the file system, instead well get the path by using the Class.getResource which will enable us to bundle ChromeDriver inside our test framework even if it is bundled into a jar file.

Basically what should be done are the following steps.

  • Determine OS type
  • Get the Chrome Driver resource and make sure it is executable using File.setExecutable(true). This is due to when packaged in a jar the execution attributes ‘x’ will be removed on Mac (and assumed on Linux too).
  • Set the “web driver.chrome.driver” system property.
  • Check that a Chrome installation exists in the default location [OPTIONAL]
private static WebDriver driver = null;
// The ChromeDriver locations under the resource folder
private static String MAC_DRIVER = "/chromedriver/mac/chromedriver";
private static String WINDOWS_DRIVER = "/chromedriver/windows/chromedriver.exe";

public static void setupChromeDriver() {
   // OS type
   if (System.getProperty("os.name").contains("Mac")) {
      File cDriver = new File(Tester.class.getResource(MAC_DRIVER).getFile());

      // Is it executable
      if (!cDriver.canExecute()) {
      System.setProperty("webdriver.chrome.driver", Tester.class.getResource(MAC_DRIVER).getFile());

      // Now checking for existence of Chrome executable.'
      if (!new File("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome").exists()) {
         throw new RuntimeException(errorMessage);
   } else {
      System.setProperty("webdriver.chrome.driver", Tester.class.getResource(WINDOWS_DRIVER).getFile());

      // Now checking for existence of Chrome executable.'
      if (!new File(System.getProperty("user.home") + "/AppData/Local/Google/Chrome/Application/chrome.exe").exists()) {
         throw new RuntimeException(errorMessage);

   ChromeOptions options = new ChromeOptions();
   driver = new ChromeDriver(options);

Test case example
Pretty straight on from here, setup WebDriver through the implemented method above and run a simple open page test to see that things worked out.

private static WebDriver driver = null;
public static void setupChromeDriver(){

public static void setupTestClass() throws Exception {

public void demoTestCase() throws Exception {

JUnit @Rule for printing test case start and end information

If you are watching the test execution trace during run-time and is missing the information about when a test cases starts and ends, then it can be easily fixed by adding a JUnit rule that prints out this information.

[TEST START] tc_shallPass
stuff printed during execution
[TEST ENDED] Time elapsed: 1.835 sec

The start and end tags gets generated automatically and hence no need to add any test case specific printing.

public void tc_shallPass() {
	System.out.println("stuff printed during execution");

To hook up the rule to all test cases in a class it has to be declared and instantiated first.

public class TestClass {
	public TestCasePrinterRule pr = new TestCasePrinterRule(System.out);
	// @Test
	// ...

So far so good, but how is this claimed to be “easily fixed” rule implemented then? Pretty straight forward by implmenting the org.junit.rules.TestRule and using a private class extending the org.junit.rules.ExternalResource. The example below calculates the execution time for the test case as well as providing the name of the current test case.

During run-time the start message will be printed before any other @Before annotated methods in the class but after the @BeforeClass methods. The apply method handles the creation of start and end tags since it has access to the test case description content, org.junit.runner.Description. The actual printing is done in the overridden before and after methods.

package com.yourcompany.customrules;

import java.io.IOException;
import java.io.OutputStream;
import java.text.DecimalFormat;

import org.junit.rules.ExternalResource;
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class TestCasePrinterRule implements TestRule {

    private OutputStream out = null;
    private final TestCasePrinter printer = new TestCasePrinter();

    private String beforeContent = null;
    private String afterContent = null;
    private long timeStart;
    private long timeEnd;
    public TestCasePrinterRule(OutputStream os) {
        out = os;

    private class TestCasePrinter extends ExternalResource {
        protected void before() throws Throwable {
            timeStart = System.currentTimeMillis();

        protected void after() {
            try {
                timeEnd = System.currentTimeMillis();
                double seconds = (timeEnd-timeStart)/1000.0;
                out.write((afterContent+"Time elapsed: "+new DecimalFormat("0.000").format(seconds)+" sec\n").getBytes());
            } catch (IOException ioe) { /* ignore */

    public final Statement apply(Statement statement, Description description) {
        beforeContent = "\n[TEST START] "+description.getMethodName()+"\n"; // description.getClassName() to get class name
        afterContent =  "[TEST ENDED] ";
        return printer.apply(statement, description);

Using JUnit @Category and Maven profiles to get stable test suites

Since it happens over and over again, no matter what project I am in. Keeping the test suites green and passing is a …

The blind eye

The one sneaking little test case that starts failing can prove to be a real killer to all automation approaches. More than once it has shown that as soon as a single test case breaks a test suite things starts to go downhill, fast. Ok, there could be acceptable resons for the test case to be broken, e.g. if there are issues in the system under test or in the test framework that are not bugs.

The usual decision taken is to leave the test case as is for now since we are all aware that it will fail for a while, BAD choice!!! All attention on the failing test suite is moving from slight attention to abandoned, and if other test in the test same test suite starts failing for various reasons it will most likely go by undetected.

What can/should be done here

  • Disable the test case and enable it again when it should be back on track
  • Maintain test suites to allow it to execute and fail but in a another suite/run to keep the MUST always pass test suites all green

The later case is the way to go.

Maven profiles to your rescue

Using Maven profiles it is easy to group your JUnit test classes and test cases in different categories. Usually this is used for grouping tests into slow, fast and browser-less tests a.s.f.. This approach can be stretched even further by using a grouping that reflects the current state of the test case. There does not have to be that many different groupable categories but at least two is needed.

  • Stable
  • Unstable/Maturing

Stable is self explained while Unstable/Maturing covers everything that is NOT always PASSING. It might be test cases that are unstable beacuse of;

  • timing issues
  • tested functionality is still under development (during sprints)
  • there is a low priority bug that will be fixed later
  • unknown issues causing it to sporadically fail
  • waiting to get a STABLE stamp

Use case 1
The last item is an approach that can be used to be really be sure that the stable test suites does not get polluted. Any new test case needs to run a set number of times (e.g. 20) in a test suite and must pass all all times before being stamped as a reliable test case.

Use case 2
When a bug surfaces that causes a test case to fail it might sometimes be the case that it wont be fixed for some time period. Instead of totally disabling or trying to fix the test case that would fail an otherwise stable test suite it shall be moved to an unstable test suite. This is a way to put the test case in a quarantine until the bug has been fixed and on the same tame make sure it still compiles and is executable.

Categorizing tests using JUnit @Category annotation
Use the JUnit @Category annotation in front of your test cases or test classes.

public void aTestCase() {
  // void

// Alternatively an unstable test case

public void anotherTestCase() {
  // void

Splitting test execution using Maven profiles

Configure your Surefire plugin to include and exclude certain test classes as well as determine a set of JUnit @Category groups to be included in the test run. Set up your pom including profiles as examplified below.


The testcase.groups property can be used to define a set of groups (JUnit @Categories) to include when tests using the actual profile.


Test suite pollution – missing term in test glossary

Test suite pollution is when one or more automated test cases deliberately are allowed to fail in a test suite. This usually leads to the executing test suite to be un-attended since errors are expected, a very unfortunate situation during development. New defects surfacing can easily be missed out and the longer the test suite gets ignored recovering it to stable can be cumbersome.

This is just an attemp to clarifying one of the ‘buzz’ terms I preach to my team. Please correct me if you think I am out of line.

Packaging and referencing images for Sikuli based tests in artifacts

This post describes one approach for packaging/bundling image resources used for Sikuli based automated test cases.

  • Bundling images as resources
  • Retrieveing/using image resources

That the images used when automating tests using Sikuli needs to versioned should be rather obvious. Naturally the majority of the pictures used in test cases reflects part of the system under test and hance shall be treated and handled as a part of the test code synchronized with what we are actually testing against. This is usually solved by having the test code in the same repository as the code under test, though it might not always be the case.

In a current test setup it was not possible to keep the test code and images together with the code under test so the test code had to be distrubited as an versioned artifact/jar-file.

Test artifact

One if the easier ways for avoiding a lot of test case maintenance is to wrap the interface to the system under test in to an artifact/API/jar-file. This allows for the artifact to be shared between different source control repositories and included in automated tests residing elsewhere (usual scenario when there are multiple trunks and components in a system). For the artifact approach the images to use for the Sikuli based tests needs to be a part of this artifact.

Bundling the images
Out of the box really, well if you are using Maven. Just put the images under resources and they gets bundled correctly in the jar. The location in the repository should be something like this.

  \- java
      \- org.project.example
  \- resources\images
      \- theImageToLookFor.PNG
      \- otherImage.PNG
      \- ...

That is all there is to it, now distribute the artifact freely, a complete pom for building a complete Sikuli test artifact including images is posted at the end.

Accessing the images in the test code
To be able to reference and use the images in your test there needs to be functionality in the artifact for getting the image resource. So implement a generic getter method that picks up all images correctly for you from inside the jar/artifact during run-time.

class TestArtifact {
	// Image file name must start with the '/' and the image name is
	// case sensitive so be thorough when using.
	private Pattern getPattern(String fileName) throws IOException {
	   BufferedImage image = ImageIO.read(getClass().getResource(fileName));
	   return new Pattern(image);

	// Example method using an image.
	// Screen screen = new Screen();
	public clickOnImage(String imageName) throws Exception {
		Match m = screen.find(getPattern(imageName));

Usage inside test cases
Note that when referencing images the path has to start with ‘/’ (fowardslash) to make sure the image is picked up correctly from within the artifact.

class Test {
	private static TestArtifact sikuli = new TestArtifact();
	public void shallClickImage() throws Exception {

Example of artifact pom

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <name>Sikili wrapper test artifact</name>



Setting up a Jenkins slave for Sikuli based tests

To be able to use the full power of Sikuli based test cases they have to work in the ‘bigger’ picture, running countinuosly in a Jenking master/slave setup. This post will go through the steps we needed to take to get Jenkins slaves (running Windows 7) to execute Sikuli based tests reliably in a larger Jenkins setup.

Covered …

  • Creating the local user
  • Disabling screen savers
  • Disabling screen lock
  • Hooking up the slave to the master

Basic requirements for a Sikuli test execution slave

To be able to use a Windows 7 based Jenkins slave for running Sikuli based tests there are a few things that needs to be understood first.

As soon as the the current user on a machine gets logged out or gets its screen locked any Sikuli based tests will stop working. Secondly to be able to run tests triggered through Jenkins the executing user (the user that runs the Jenkins process) needs to have access to the desktop to be able to run applications like browsers in a visible way. Running Jenkins as a web service in Windows 7 will prohibit visible execution of applications or in other words the service is not allowed to access the desktop. Meaning that things are not as simple as starting Jenkins as a Windows service and then hook it up to the Jenkins master. Java web start (JNLP) is the way to go.

Requirements in short …

  • local user that is always logged on
  • machine that never gets locked
  • node that connects to the Jenkins master using Java web start (JNLP)
  • Java and environment variables configured properly to run Sikuli-based test cases (here)

Getting the local user in place

1) Create a local superuser account on the slave node.

2) Set the following keys in the registry to ensure that the user never gets logged out.

Open the registry editor (regedit) and add a new DWORD as named DisableLockWorkstation and give it the value 1 (1 = disable) to the path below.


3) It also has to be made sure that the local user always gets logged on automatically if the physical machine gets restarted. Open the following entry in the registry editor.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon

Make sure the following keys are present and that they are set accordingly.

AutoAdminLogon Value : 1
DefaultPassword Value : 'the password set for the created local super user'
DefaultUsername Value : 'the super user name'

Reboot to try it out.

Disabling all screen savers

One again using the registry, set the ScreenSaveActive key to 0 in the location below.

HKEY_CURRENT_USER\Control Panel\Desktop

Connecting to Jenkins master

1) On the test execution slave, open any browser that has java support
2) Navigate to the URL of the Jenkins master
3) Manage Jenkins -> Manage Nodes
4) Add a new node
5) Select the new node and click on the Launch button to connect to the master using JNLP, if the node already has all the correct Java installations and environment variables we are set to go (here).

JNLP connection launch button as seen in Jenkins


Firewalls, getting the JNLP connection to work might need changes in firewalls. Jenkins is by default configured to use a randomly generated port for the JNLP connection. Change this setting to use a fixed port number then there should not be to big of a trouble getting the firewall rules in place.

Manage Jenkins -> Configure System