Packaging and referencing images for Sikuli based tests in artifacts

This post describes one approach for packaging/bundling image resources used for Sikuli based automated test cases.

  • Bundling images as resources
  • Retrieveing/using image resources

That the images used when automating tests using Sikuli needs to versioned should be rather obvious. Naturally the majority of the pictures used in test cases reflects part of the system under test and hance shall be treated and handled as a part of the test code synchronized with what we are actually testing against. This is usually solved by having the test code in the same repository as the code under test, though it might not always be the case.

In a current test setup it was not possible to keep the test code and images together with the code under test so the test code had to be distrubited as an versioned artifact/jar-file.

Test artifact

One if the easier ways for avoiding a lot of test case maintenance is to wrap the interface to the system under test in to an artifact/API/jar-file. This allows for the artifact to be shared between different source control repositories and included in automated tests residing elsewhere (usual scenario when there are multiple trunks and components in a system). For the artifact approach the images to use for the Sikuli based tests needs to be a part of this artifact.

Bundling the images
Out of the box really, well if you are using Maven. Just put the images under resources and they gets bundled correctly in the jar. The location in the repository should be something like this.

  \- java
      \- org.project.example
  \- resources\images
      \- theImageToLookFor.PNG
      \- otherImage.PNG
      \- ...

That is all there is to it, now distribute the artifact freely, a complete pom for building a complete Sikuli test artifact including images is posted at the end.

Accessing the images in the test code
To be able to reference and use the images in your test there needs to be functionality in the artifact for getting the image resource. So implement a generic getter method that picks up all images correctly for you from inside the jar/artifact during run-time.

class TestArtifact {
	// Image file name must start with the '/' and the image name is
	// case sensitive so be thorough when using.
	private Pattern getPattern(String fileName) throws IOException {
	   BufferedImage image =;
	   return new Pattern(image);

	// Example method using an image.
	// Screen screen = new Screen();
	public clickOnImage(String imageName) throws Exception {
		Match m = screen.find(getPattern(imageName));;

Usage inside test cases
Note that when referencing images the path has to start with ‘/’ (fowardslash) to make sure the image is picked up correctly from within the artifact.

class Test {
	private static TestArtifact sikuli = new TestArtifact();
	public void shallClickImage() throws Exception {

Example of artifact pom

<project xmlns="" xmlns:xsi=""

    <name>Sikili wrapper test artifact</name>



Using Sikuli to test legacy Flash

With loads of legacy Flash/Flex code where test automation support is limited or in most cases even non-existing there is only so much that can be done in forms of automation. I cannot understand why I did not pick up Sikuli earlier but I should have, that is all I am saying. After two days of playing around and demoing QA engineers around me are so eager to get going.

The concept of image recognition and test case maintenance does not sound too tempting but for the case with legacy flash implementations where the graphics will surely not change it would be unfair not to try it out. So we did but it was not out of the box.

The simplest of test cases
In short, unless you want to read it from Project Sikuli. Sikuli can match a predefined image to a section of the screen that is VERY similar to predefined image, ‘best fit’. So for example if you take a small screenshot of an object in you Flash implementation and save it as an image Sikuli can then be used to interact with this object, clicking on it, dragging it, checking if it can find it on the page asf..

Sikuli can rather easily be tested out, for example against a common Flash application like the one in the clip below. To get a test cases running just crop out the correct pictures from the application under test.

The pictures below is found in the application under test without any problems for Sikuli.

Playing roulette using Sikuli.

The Junit test flow looks like this.

// Java
// Example Sikuli Test flow, no verifications in place it only reflects what is
// visible in the youtube clip above.
public void playRoulette() throws Exception {
  SikuliDemo sikuliDemo = new SikuliDemo();

Integrating into your Java test framework
So how to, using Java? In fact code wise, very little code is needed to get the basic functionality in place, click and verify. The code sample below simply looks for an image snapshotted from the application under test anywhere on the screen and clicks it as well as checking for the visibility of another. Just make sure that sikuli-script.jar is in the classpath.

// Java
import org.sikuli.script.Screen;

public class SikuliDemo {
  public boolean clickButton(String gameName, String imageName) {
    // Window is already opened by WebDriver
    Screen s = new Screen();
    try {
      s.exists(imageName, FLASH_TIMEOUT);, FLASH_TIMEOUT);
    } catch (FindFailed ff) {
      return false;
    return true;

  public boolean verifyImage(String gameName, String imageName) {
    // Window is already opened by WebDriver
    Screen s = new Screen();
    try {
      s.wait(imageName, FLASH_TIMEOUT);
    } catch (FindFailed ff) {
      return false;
    return true;

Getting things to work
… not as easy as expected, the following issues are some that surfaced.

Running on a 64bit Windows system

Be thorough when configuring your system or it is very easy to get error messages relating to Win32Util.dll or VisionProxy.dll. I basically had to uninstall all previously installed Java versions to get it working.

A Java SDK/JDK 32bit installation is needed, these end with ‘i586’ on the Oracle download site. Then make sure the following environment variables are set accordingly on the running system.

  • SIKULI_HOME – Sikuli installation folder
  • JAVA_HOME – To the 32bit installation
  • PATH – Add %SIKULI_HOME%\libs;JAVA_HOME\bin

Running from Jenkins
The next obstacle, Sikuli needs access to the desktop to work. The most common way to run Jenkins is a Windows web service, yes it can be configured to get access to the desktop but no matter how we could not get it working properly. The solution that worked out in the end was to run Jenkins standalone from a command window using;

$ java -jar jenkins.war

By doing that Jenkins had access to the desktop as well as making it possible to watch the tests execute when logged in to the build agent.

Firefox in full screen
Some tests still had problems running smoothly on some machines due to screen resolutions and visibility. Since the Flash was embedded into a web page setting Firefox to expand to full screen was a fix that worked out quite smoothly. Using the WebDriver and to get the FirefoxDriver simulate an F11 key press was done with the single line of code below.

// Java


Extending WebTest for FitNesse and Selenium

WebTest is Selenium wrapper used in FitNesse test cases. It contains a number of methods that makes it easy to write GUI based test. However it only supports a limited number of methods (WebTest reference) and in most cases there is a need for tweaking these or simply adding new ones.

Extending the com.neuri.webfixture.WebTest class should be a simple thing to do if it had not been for the private Selenium instance attribute.

This short post will describe an approach for how to be able to extend the com.neuri.webfixture.WebTest class, the example is based on webtestfixtures version 2.01.

Using reflection to get the selenium instance

For all calls to the Selenium RC API to work the Selenium instance hidden in the super class needs to be used. This instance can only be fetched using reflection. Add the method below to your extending class and you will get access to the Selenium instance simply by calling getSeleniumInstance()

protected Selenium getSeleniumInstance() {
  try {
    Field privateInstance = com.neuri.webfixture.WebTest.class.getDeclaredField("instance");
    Selenium seleniumInstance = (Selenium) privateInstance.get(this);
    return seleniumInstance;
  } catch (NoSuchFieldException ndfe) {
    // Do nothing!
  } catch (IllegalAccessException iae) {
    // Do nothing!
  return null;

Usage example
Retrieve the Selenium instance and you have access to the Selenium RC API, in the example below getTitle().

public class UnitTHWebTest extends com.neuri.webfixture.WebTest {
  protected Selenium getSeleniumInstance() {...}

  public boolean pageContainsTitleText(String text) {
    Selenium si = getSeleniumInstance();
    if (si.getTitle().equalsIgnoreCase(text)) {
      return true;
    } else {
      return false;

Download from:

soapUI inline request parameterization made simple using static Java methods

Maintainability and reuse when using a lot of Groovy scripting or in-house Java libs with the free version of soapUI can be really cumbersome. The same groovy code snippets are all over the place, it does not matter if a lot of core functionality is mapped into the java libraries there are still some Groovy code to be written to get this to work in the long run.

I recently had to struggle creating maintainable test suites covering more than 200 test cases and in total easily more than a thousand different requests. There was a clear need for making the test suites easy to maintain and extend. Removing all the duplicated Groovy code was the top priority. Blow is the usual scenario we had for the majority of our tests in the beginning.

A) Groovy script

  • importing any bin/ext java libraries
  • invoking a given method the e.g. creates a timestamp value
  • put the generated timestamp into a test case property

B) Test request

  • inline expansion of the test case property

This was really an overkill approach when dealing with hundreds and hundreds of different test requests. So how to get this in to small one-liners inside the requests directly without even having to deal with test case properties asf..

Using static Java methods

Just implement a set of static Java methods and inline these in your test requests. No need for library imports or instance creation just call the method needed with arguments that in turn can be dynamically created.

Below is an example class that can be added to a jar file put in the soapUIinstallation/bin/ext folder.

public class TimeStampCreator {
  public static String getToday() { ... }
  public static String getTodayPlusDays(int days) { ... }
  public static String getTodayMinusDays(int days) { ... }

The methods above can be called accordingly as described in the TestRequest (SOAP) example below, note the nested test case parameter expansion.

<args xsi:type="api:ArrayOf_xsd_string" soapenc:arrayType="xsd:string[]">

Works like a charm…