Packaging and referencing images for Sikuli based tests in artifacts

This post describes one approach for packaging/bundling image resources used for Sikuli based automated test cases.

  • Bundling images as resources
  • Retrieveing/using image resources

That the images used when automating tests using Sikuli needs to versioned should be rather obvious. Naturally the majority of the pictures used in test cases reflects part of the system under test and hance shall be treated and handled as a part of the test code synchronized with what we are actually testing against. This is usually solved by having the test code in the same repository as the code under test, though it might not always be the case.

In a current test setup it was not possible to keep the test code and images together with the code under test so the test code had to be distrubited as an versioned artifact/jar-file.

Test artifact

One if the easier ways for avoiding a lot of test case maintenance is to wrap the interface to the system under test in to an artifact/API/jar-file. This allows for the artifact to be shared between different source control repositories and included in automated tests residing elsewhere (usual scenario when there are multiple trunks and components in a system). For the artifact approach the images to use for the Sikuli based tests needs to be a part of this artifact.

Bundling the images
Out of the box really, well if you are using Maven. Just put the images under resources and they gets bundled correctly in the jar. The location in the repository should be something like this.

  \- java
      \- org.project.example
  \- resources\images
      \- theImageToLookFor.PNG
      \- otherImage.PNG
      \- ...

That is all there is to it, now distribute the artifact freely, a complete pom for building a complete Sikuli test artifact including images is posted at the end.

Accessing the images in the test code
To be able to reference and use the images in your test there needs to be functionality in the artifact for getting the image resource. So implement a generic getter method that picks up all images correctly for you from inside the jar/artifact during run-time.

class TestArtifact {
	// Image file name must start with the '/' and the image name is
	// case sensitive so be thorough when using.
	private Pattern getPattern(String fileName) throws IOException {
	   BufferedImage image =;
	   return new Pattern(image);

	// Example method using an image.
	// Screen screen = new Screen();
	public clickOnImage(String imageName) throws Exception {
		Match m = screen.find(getPattern(imageName));;

Usage inside test cases
Note that when referencing images the path has to start with ‘/’ (fowardslash) to make sure the image is picked up correctly from within the artifact.

class Test {
	private static TestArtifact sikuli = new TestArtifact();
	public void shallClickImage() throws Exception {

Example of artifact pom

<project xmlns="" xmlns:xsi=""

    <name>Sikili wrapper test artifact</name>



Setting up a Jenkins slave for Sikuli based tests

To be able to use the full power of Sikuli based test cases they have to work in the ‘bigger’ picture, running countinuosly in a Jenking master/slave setup. This post will go through the steps we needed to take to get Jenkins slaves (running Windows 7) to execute Sikuli based tests reliably in a larger Jenkins setup.

Covered …

  • Creating the local user
  • Disabling screen savers
  • Disabling screen lock
  • Hooking up the slave to the master

Basic requirements for a Sikuli test execution slave

To be able to use a Windows 7 based Jenkins slave for running Sikuli based tests there are a few things that needs to be understood first.

As soon as the the current user on a machine gets logged out or gets its screen locked any Sikuli based tests will stop working. Secondly to be able to run tests triggered through Jenkins the executing user (the user that runs the Jenkins process) needs to have access to the desktop to be able to run applications like browsers in a visible way. Running Jenkins as a web service in Windows 7 will prohibit visible execution of applications or in other words the service is not allowed to access the desktop. Meaning that things are not as simple as starting Jenkins as a Windows service and then hook it up to the Jenkins master. Java web start (JNLP) is the way to go.

Requirements in short …

  • local user that is always logged on
  • machine that never gets locked
  • node that connects to the Jenkins master using Java web start (JNLP)
  • Java and environment variables configured properly to run Sikuli-based test cases (here)

Getting the local user in place

1) Create a local superuser account on the slave node.

2) Set the following keys in the registry to ensure that the user never gets logged out.

Open the registry editor (regedit) and add a new DWORD as named DisableLockWorkstation and give it the value 1 (1 = disable) to the path below.


3) It also has to be made sure that the local user always gets logged on automatically if the physical machine gets restarted. Open the following entry in the registry editor.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon

Make sure the following keys are present and that they are set accordingly.

AutoAdminLogon Value : 1
DefaultPassword Value : 'the password set for the created local super user'
DefaultUsername Value : 'the super user name'

Reboot to try it out.

Disabling all screen savers

One again using the registry, set the ScreenSaveActive key to 0 in the location below.

HKEY_CURRENT_USER\Control Panel\Desktop

Connecting to Jenkins master

1) On the test execution slave, open any browser that has java support
2) Navigate to the URL of the Jenkins master
3) Manage Jenkins -> Manage Nodes
4) Add a new node
5) Select the new node and click on the Launch button to connect to the master using JNLP, if the node already has all the correct Java installations and environment variables we are set to go (here).

JNLP connection launch button as seen in Jenkins


Firewalls, getting the JNLP connection to work might need changes in firewalls. Jenkins is by default configured to use a randomly generated port for the JNLP connection. Change this setting to use a fixed port number then there should not be to big of a trouble getting the firewall rules in place.

Manage Jenkins -> Configure System

Using Sikuli to test legacy Flash

With loads of legacy Flash/Flex code where test automation support is limited or in most cases even non-existing there is only so much that can be done in forms of automation. I cannot understand why I did not pick up Sikuli earlier but I should have, that is all I am saying. After two days of playing around and demoing QA engineers around me are so eager to get going.

The concept of image recognition and test case maintenance does not sound too tempting but for the case with legacy flash implementations where the graphics will surely not change it would be unfair not to try it out. So we did but it was not out of the box.

The simplest of test cases
In short, unless you want to read it from Project Sikuli. Sikuli can match a predefined image to a section of the screen that is VERY similar to predefined image, ‘best fit’. So for example if you take a small screenshot of an object in you Flash implementation and save it as an image Sikuli can then be used to interact with this object, clicking on it, dragging it, checking if it can find it on the page asf..

Sikuli can rather easily be tested out, for example against a common Flash application like the one in the clip below. To get a test cases running just crop out the correct pictures from the application under test.

The pictures below is found in the application under test without any problems for Sikuli.

Playing roulette using Sikuli.

The Junit test flow looks like this.

// Java
// Example Sikuli Test flow, no verifications in place it only reflects what is
// visible in the youtube clip above.
public void playRoulette() throws Exception {
  SikuliDemo sikuliDemo = new SikuliDemo();

Integrating into your Java test framework
So how to, using Java? In fact code wise, very little code is needed to get the basic functionality in place, click and verify. The code sample below simply looks for an image snapshotted from the application under test anywhere on the screen and clicks it as well as checking for the visibility of another. Just make sure that sikuli-script.jar is in the classpath.

// Java
import org.sikuli.script.Screen;

public class SikuliDemo {
  public boolean clickButton(String gameName, String imageName) {
    // Window is already opened by WebDriver
    Screen s = new Screen();
    try {
      s.exists(imageName, FLASH_TIMEOUT);, FLASH_TIMEOUT);
    } catch (FindFailed ff) {
      return false;
    return true;

  public boolean verifyImage(String gameName, String imageName) {
    // Window is already opened by WebDriver
    Screen s = new Screen();
    try {
      s.wait(imageName, FLASH_TIMEOUT);
    } catch (FindFailed ff) {
      return false;
    return true;

Getting things to work
… not as easy as expected, the following issues are some that surfaced.

Running on a 64bit Windows system

Be thorough when configuring your system or it is very easy to get error messages relating to Win32Util.dll or VisionProxy.dll. I basically had to uninstall all previously installed Java versions to get it working.

A Java SDK/JDK 32bit installation is needed, these end with ‘i586’ on the Oracle download site. Then make sure the following environment variables are set accordingly on the running system.

  • SIKULI_HOME – Sikuli installation folder
  • JAVA_HOME – To the 32bit installation
  • PATH – Add %SIKULI_HOME%\libs;JAVA_HOME\bin

Running from Jenkins
The next obstacle, Sikuli needs access to the desktop to work. The most common way to run Jenkins is a Windows web service, yes it can be configured to get access to the desktop but no matter how we could not get it working properly. The solution that worked out in the end was to run Jenkins standalone from a command window using;

$ java -jar jenkins.war

By doing that Jenkins had access to the desktop as well as making it possible to watch the tests execute when logged in to the build agent.

Firefox in full screen
Some tests still had problems running smoothly on some machines due to screen resolutions and visibility. Since the Flash was embedded into a web page setting Firefox to expand to full screen was a fix that worked out quite smoothly. Using the WebDriver and to get the FirefoxDriver simulate an F11 key press was done with the single line of code below.

// Java