JUnit: Excluding data driven tests

Once in a while there is that single test case that cannot be executed for certain data when running data driven tests. This could be a true  annoyment and it is tempting to break out those tests into a separate test class, don’t do that and stay away from taking shortcuts that will give you;

  • False test statistics, reporting passing test that has never been run
  • Avoid Ignoring tests e.g. using conditional ignores or JUnit’s assume.

A simple way to exclude a test from being executed when running data driven tests using the Parameterized.class is to set up a conditional JUnit rule. A test rule that based on the current test data in the data driven loop runs OR excludes a specifc test.

Consider the scenario below

We run the all tests below with the following samples “a”, “b”, “c” but know that test02 will not work for the sample “b” so we will have to deal with that in a clever way.

@Runwith(Parameterized.class)
public class Test {

  private String sample;

  public Test(String sample) {
    this.sample = sample;
  }

  @Parameters
  public static Collection<Object[]> generateSamples() {
    final List <Object[]> samples = new ArrayList<Object[]>();
    sample.add(new Object[]{"a"});
    sample.add(new Object[]{"b"});
    sample.add(new Object[]{"c"});
    return samples;
  }

  @Test
  public void test01() {
    // Works with sample "a", "b", "c"
  }

  @Test
  public void test02() {
    // Works with sample "a", "c" BUT NOT "b"
  }

  @Test
  public void test03() {
    // Works with sample "a", "b", "c"
  }
}

Mark-up with test annotation

Step one is to mark-up a test case so that we know that it will only run for certain samples. We will do this by adding an annotation, in this case we call it Samples.

@Test
public void test01() { ... }

@Samples({"a","c"})
@Test
public void test02() { ... }

@Samples({"a","b","c"})
@Test
public void test03() { ... }

Using the test annotation Samples we can start filtering during run-time if to execute a test case or not. A test without the annotation will always run the test no matter what data that is reurned in the data driven loop. If the annotation is in place the test will only run for the samples that matches the values in the annotation. In case of test02 it will only run for samples “a” and “c”.

The test annotation

Adding the annotation is straight forward, read up on it here.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Samples {
    String [] value();
}

Adding the exclusion rule

Worth knowing is that JUnit rules will always be created before a test class constructor. This means that a rule will not know the test data sample of the current data driven iteration. The current data driven value is always passed to the test class constructor (Parameterized.class) and hence the rule we are going to implement needs to get that information as well at that point.

@Runwith(Parameterized.class)
public class Test {

  @Rule
  public OnlyRunForSampleRule rule = new OnlyRunForSampleRule();
  private String sample;

  public Test(String sample) {
    this.sample = sample;
    rule.setSample(sample); // <<-- HERE
  }

With the current data sample and the Samples annotation values known at run-time, we define a simple rule that excludes test cases if they are not supposed to run for certain test data.

public class OnlyRunForSampleRule implements TestRule {

  private String sample;

  @Override
  public Statement apply(Statement s, Description d) {
    Protocols annotation = d.getAnnotation(Samples.class);
    // No annotation/samples matching, always run
    if (annotation == null) {
      return s;
    }
    // Match! One sample value matches current parameterized sample value
    else if (Arrays.asList(annotation.value()).contains(protocol)) {
      return s;
    }
    // No match in the samples annotation, skipping
    return new Statement() {
      @Override
      public void evaluate() throws Throwable {}
    };
  }

  public void setSample(String sample) {
    this.sample = sample;
  }
}

The rule above is defined to run tests on known samples, creating the inverse of this rule is simply done by returning the empty Statement in the else if that checks for matches in the Samples annotation.

Debugging WireMock calls when using JUnit WireMockRule

Mocking using the WireMockRule in your JUnit test classes and struggle with 404’s?

It is not that trivial to find in the WireMock documentation but it is in there, under ‘Listening for requests’ @ http://wiremock.org/verifying.html. Plain debugging fine, but sometimes one really wants to know the details of the calls made to the underlying services that are consumed, especially when WireMocking these services and there is a fine grained matching mechanism to deal with.

Below is the quick awesome tip to get the details you need to resolve the WireMock returned 404’s easily.

Add a request listener to your WireMockRule and Use Java 8 lambdas to smoothly implement the WireMock interface RequestListener that has the single method requestReceived(Request request, Response response). Print out the reponse and request details you want. Run your tests and check the print outs, all set!

import com.github.tomakehurst.wiremock.junit.WireMockRule;

public class Test {

   @Rule
   public WireMockRule wireMockRule = new WireMockRule(6969);

   @Before
   public void setupTest() {
      wireMockRule.addMockServiceRequestListener((request, response) -> {
         System.out.println("URL Requested => " + request.getAbsoluteUrl());
         System.out.println("Request Body => " + request.getBodyAsString());
         System.out.println("Request Headers => " + request.getAllHeaderKeys());
         System.out.println("Response Status => " + response.getStatus());
         System.out.println("Response Body => " + response.getBodyAsString());
      });
   }
   ...
}

Test automation don’ts #1 – Separate repositories

Since this is one of these days when frustration is at what feels like an all time high ‘again’ what better to do that to get it all out of you. When it comes to test automation there are so many poor decisions out there… yes, many. I’ll try to address some of these here. The first one out is, da-da.

#1 Do not keep automation code in a separate reposity

So what does it mean, consider the case where all test cases and test framework code is in a nicely well structured repository and the code to be tested resides in an equally professionally handled repository. But what happens when it is time to run test cases? Is the HEAD of these repositories/branches ‘always’ in synch, well most likely they are not, not even if the codebase is completely ‘branch free’. The end result, most likely broken tests that will cause a lot of waste.

So for the case where there is one repository for the product code, DO NOT put the automation code in a sperate repository or you’ll end up struggling with failing test cases as soon as the branches get out of synch. On top of this guess what kind of overhead you will face if a team/teams are working with a feature branch strategy, let me smile and mention that this is what I struggle with today broken tests all over the place and no one has a clue about which failing tests that are regressions and which ones are expected to fail?

I have seen the scenario with separate repositories at several places and the usual reason is that the system (under test) it self is spread out over multiple repositories and the aim was to keep the automated tests in one place supporting all parts of a ‘system’. Not even tests that are defined as end-2-end tests spanning a full system will handle the scenario smoothly. I have been involved in several approaches trying to resolve problems without putting the automation code in the same repository, none of the approaches have been fully successful.

Identified problems

  • Test code out of synch with code under test
  • TWICE the amount of branches to maintain and work with

Bundling ChromeDriver with your test code

This post will just exemplify one way of maintaining the chrome drivers used for running automated tests with WebDriver and the Chrome browser without having to update and install the ChromeDriver to all possible nodes where tests will be running. The ChromeDriver will simply be bundled with the running tests and put under source control as for all other test ware.

Downloading links
http://code.google.com/p/selenium/wiki/ChromeDriver

Bundling
Start by putting the the drivers under the resource folder so that they will be picked up by Maven per default.

<project root>/src/main/resources/chromedriver/mac/chromedriver
<project root>/src/main/resources/chromedriver/windows/chromedriver.exe

Implementation
There are of course different drivers for different OS types and this needs to be handled using the os.name system property.

As per the ChromeDriver usage instructions (here) a system property has to be set pointing to the ChromeDriver server to use for the Chrome browser bridging. We will not point to a fixed location in the file system, instead well get the path by using the Class.getResource which will enable us to bundle ChromeDriver inside our test framework even if it is bundled into a jar file.

Basically what should be done are the following steps.

  • Determine OS type
  • Get the Chrome Driver resource and make sure it is executable using File.setExecutable(true). This is due to when packaged in a jar the execution attributes ‘x’ will be removed on Mac (and assumed on Linux too).
  • Set the “web driver.chrome.driver” system property.
  • Check that a Chrome installation exists in the default location [OPTIONAL]
private static WebDriver driver = null;
// The ChromeDriver locations under the resource folder
private static String MAC_DRIVER = "/chromedriver/mac/chromedriver";
private static String WINDOWS_DRIVER = "/chromedriver/windows/chromedriver.exe";

public static void setupChromeDriver() {
   // OS type
   if (System.getProperty("os.name").contains("Mac")) {
      File cDriver = new File(Tester.class.getResource(MAC_DRIVER).getFile());

      // Is it executable
      if (!cDriver.canExecute()) {
         cDriver.setExecutable(true);
      }
      System.setProperty("webdriver.chrome.driver", Tester.class.getResource(MAC_DRIVER).getFile());

      // Now checking for existence of Chrome executable.'
      if (!new File("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome").exists()) {
         throw new RuntimeException(errorMessage);
      }
   } else {
      System.setProperty("webdriver.chrome.driver", Tester.class.getResource(WINDOWS_DRIVER).getFile());

      // Now checking for existence of Chrome executable.'
      if (!new File(System.getProperty("user.home") + "/AppData/Local/Google/Chrome/Application/chrome.exe").exists()) {
         throw new RuntimeException(errorMessage);
      }
   }

   ChromeOptions options = new ChromeOptions();
   options.addArguments("--start-maximized");
   options.addArguments("--ignore-certificate-errors");
   driver = new ChromeDriver(options);
}

Test case example
Pretty straight on from here, setup WebDriver through the implemented method above and run a simple open page test to see that things worked out.

private static WebDriver driver = null;
public static void setupChromeDriver(){
   ...
}

@BeforeClass
public static void setupTestClass() throws Exception {
   setupChromeDriver();
}

@Test
public void demoTestCase() throws Exception {
   driver.get("http://code.google.com/p/selenium/wiki/ChromeDriver");
   Thread.sleep(1000);
}

Packaging and referencing images for Sikuli based tests in artifacts

This post describes one approach for packaging/bundling image resources used for Sikuli based automated test cases.

  • Bundling images as resources
  • Retrieveing/using image resources

That the images used when automating tests using Sikuli needs to versioned should be rather obvious. Naturally the majority of the pictures used in test cases reflects part of the system under test and hance shall be treated and handled as a part of the test code synchronized with what we are actually testing against. This is usually solved by having the test code in the same repository as the code under test, though it might not always be the case.

In a current test setup it was not possible to keep the test code and images together with the code under test so the test code had to be distrubited as an versioned artifact/jar-file.

Test artifact

One if the easier ways for avoiding a lot of test case maintenance is to wrap the interface to the system under test in to an artifact/API/jar-file. This allows for the artifact to be shared between different source control repositories and included in automated tests residing elsewhere (usual scenario when there are multiple trunks and components in a system). For the artifact approach the images to use for the Sikuli based tests needs to be a part of this artifact.

Bundling the images
Out of the box really, well if you are using Maven. Just put the images under resources and they gets bundled correctly in the jar. The location in the repository should be something like this.


src\main
  \- java
      \- org.project.example
  \- resources\images
      \- theImageToLookFor.PNG
      \- otherImage.PNG
      \- ...

That is all there is to it, now distribute the artifact freely, a complete pom for building a complete Sikuli test artifact including images is posted at the end.

Accessing the images in the test code
To be able to reference and use the images in your test there needs to be functionality in the artifact for getting the image resource. So implement a generic getter method that picks up all images correctly for you from inside the jar/artifact during run-time.

class TestArtifact {
	// Image file name must start with the '/' and the image name is
	// case sensitive so be thorough when using.
	private Pattern getPattern(String fileName) throws IOException {
	   BufferedImage image = ImageIO.read(getClass().getResource(fileName));
	   return new Pattern(image);
	}

	// Example method using an image.
	// Screen screen = new Screen();
	public clickOnImage(String imageName) throws Exception {
		Match m = screen.find(getPattern(imageName));
		screen.click(m);
	}
}

Usage inside test cases
Note that when referencing images the path has to start with ‘/’ (fowardslash) to make sure the image is picked up correctly from within the artifact.

class Test {
	private static TestArtifact sikuli = new TestArtifact();
	
	@Test
	public void shallClickImage() throws Exception {
		sikuli.clickOnImage("/images/theImageToLookFor.PNG");
	}
}

Example of artifact pom

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.yourcompany.automation.apis</groupId>
    <artifactId>sikuliwrapper</artifactId>
    <packaging>jar</packaging>
    <name>Sikili wrapper test artifact</name>
    <version>1.0.0-SNAPSHOT</version>
    <inceptionYear>2012</inceptionYear>

    <dependencies>
        <dependency>
            <groupId>org.sikuli</groupId>
            <artifactId>sikuli-script</artifactId>
            <version>0.10.2</version>
            <type>jar</type>
        </dependency>
    </dependencies>

</project>

Using Sikuli to test legacy Flash

With loads of legacy Flash/Flex code where test automation support is limited or in most cases even non-existing there is only so much that can be done in forms of automation. I cannot understand why I did not pick up Sikuli earlier but I should have, that is all I am saying. After two days of playing around and demoing QA engineers around me are so eager to get going.

The concept of image recognition and test case maintenance does not sound too tempting but for the case with legacy flash implementations where the graphics will surely not change it would be unfair not to try it out. So we did but it was not out of the box.

The simplest of test cases
In short, unless you want to read it from Project Sikuli. Sikuli can match a predefined image to a section of the screen that is VERY similar to predefined image, ‘best fit’. So for example if you take a small screenshot of an object in you Flash implementation and save it as an image Sikuli can then be used to interact with this object, clicking on it, dragging it, checking if it can find it on the page asf..

Sikuli can rather easily be tested out, for example against a common Flash application like the one in the clip below. To get a test cases running just crop out the correct pictures from the application under test.

The pictures below is found in the application under test without any problems for Sikuli.

Playing roulette using Sikuli.

The Junit test flow looks like this.


// Java
//
// Example Sikuli Test flow, no verifications in place it only reflects what is
// visible in the youtube clip above.
//
@Test
public void playRoulette() throws Exception {
  SikuliDemo sikuliDemo = new SikuliDemo();
  sikuliDemo.clickButton("bet5.PNG");
  Thread.sleep(10000);
  sikuliDemo.clickButton("betonred.PNG");
  sikuliDemo.clickButton("bet5.PNG");
  sikuliDemo.clickButton("betonred.PNG");
  Thread.sleep(10000);
  sikuliDemo.clickButton("cleartable.PNG");
  Thread.sleep(10000);
  sikuliDemo.clickButton("betonodd.PNG");
  sikuliDemo.clickButton("bet5.PNG");
  sikuliDemo.clickButton("betonodd.PNG");
  Thread.sleep(10000);
  sikuliDemo.clickButton("bet25.PNG");
  Thread.sleep(10000);
  sikuliDemo.clickButton("beton13.PNG");
  Thread.sleep(10000);
  sikuliDemo.clickButton("spin.PNG");
  Thread.sleep(300000);
}

Integrating into your Java test framework
So how to, using Java? In fact code wise, very little code is needed to get the basic functionality in place, click and verify. The code sample below simply looks for an image snapshotted from the application under test anywhere on the screen and clicks it as well as checking for the visibility of another. Just make sure that sikuli-script.jar is in the classpath.


// Java
import org.sikuli.script.Screen;

public class SikuliDemo {
  public boolean clickButton(String gameName, String imageName) {
    // Window is already opened by WebDriver
    Screen s = new Screen();
    try {
      s.exists(imageName, FLASH_TIMEOUT);
      s.click(imageName, FLASH_TIMEOUT);
    } catch (FindFailed ff) {
      return false;
    }
    return true;
  }

  public boolean verifyImage(String gameName, String imageName) {
    // Window is already opened by WebDriver
    Screen s = new Screen();
    try {
      s.wait(imageName, FLASH_TIMEOUT);
    } catch (FindFailed ff) {
      return false;
    }
    return true;
  }
}

Getting things to work
… not as easy as expected, the following issues are some that surfaced.

Running on a 64bit Windows system

Be thorough when configuring your system or it is very easy to get error messages relating to Win32Util.dll or VisionProxy.dll. I basically had to uninstall all previously installed Java versions to get it working.

A Java SDK/JDK 32bit installation is needed, these end with ‘i586’ on the Oracle download site. Then make sure the following environment variables are set accordingly on the running system.

  • SIKULI_HOME – Sikuli installation folder
  • JAVA_HOME – To the 32bit installation
  • PATH – Add %SIKULI_HOME%\libs;JAVA_HOME\bin

Running from Jenkins
The next obstacle, Sikuli needs access to the desktop to work. The most common way to run Jenkins is a Windows web service, yes it can be configured to get access to the desktop but no matter how we could not get it working properly. The solution that worked out in the end was to run Jenkins standalone from a command window using;

$ java -jar jenkins.war

By doing that Jenkins had access to the desktop as well as making it possible to watch the tests execute when logged in to the build agent.

Firefox in full screen
Some tests still had problems running smoothly on some machines due to screen resolutions and visibility. Since the Flash was embedded into a web page setting Firefox to expand to full screen was a fix that worked out quite smoothly. Using the WebDriver and to get the FirefoxDriver simulate an F11 key press was done with the single line of code below.

// Java
((FirefoxDriver)driver).getKeyboard().pressKey(Keys.F11);

Downloads

Extending WebTest for FitNesse and Selenium

WebTest is Selenium wrapper used in FitNesse test cases. It contains a number of methods that makes it easy to write GUI based test. However it only supports a limited number of methods (WebTest reference) and in most cases there is a need for tweaking these or simply adding new ones.

Extending the com.neuri.webfixture.WebTest class should be a simple thing to do if it had not been for the private Selenium instance attribute.

This short post will describe an approach for how to be able to extend the com.neuri.webfixture.WebTest class, the example is based on webtestfixtures version 2.01.

Using reflection to get the selenium instance

For all calls to the Selenium RC API to work the Selenium instance hidden in the super class needs to be used. This instance can only be fetched using reflection. Add the method below to your extending class and you will get access to the Selenium instance simply by calling getSeleniumInstance()


protected Selenium getSeleniumInstance() {
  try {
    Field privateInstance = com.neuri.webfixture.WebTest.class.getDeclaredField("instance");
    privateInstance.setAccessible(true);
    Selenium seleniumInstance = (Selenium) privateInstance.get(this);
    return seleniumInstance;
  } catch (NoSuchFieldException ndfe) {
    // Do nothing!
  } catch (IllegalAccessException iae) {
    // Do nothing!
  }
  return null;
}

Usage example
Retrieve the Selenium instance and you have access to the Selenium RC API, in the example below getTitle().


public class UnitTHWebTest extends com.neuri.webfixture.WebTest {
  protected Selenium getSeleniumInstance() {...}

  public boolean pageContainsTitleText(String text) {
    Selenium si = getSeleniumInstance();
    if (si.getTitle().equalsIgnoreCase(text)) {
      return true;
    } else {
      return false;
    }
  }
}

Download from: