ScalaTests on Jenkins using Maven, use case examples

This is a post for how to do it with Maven, the same functionality can be acheived by using Sbt if preferring that option.

There are a number of threads out there that addresses the topic of configuring and running ScalaTest test cases/specs on Jenkins, some threads older than others but since this seems to be an re-occuring question topic why not to examplify how it can be done based for some of the most common use cases that seems to be popping up in the forums. So if you have problems setting up jobs that are running specific test specs or test that are tagged then have a look at the examples below.

Core knowledge basics

scalatest-maven-plugin

This plugins enbles execution of ScalaTests tests using Maven without any extra fuzz like using @RunWith(classOf[JUnitRunner])… I takes a set of configuration options that are well defined on the . So start by adding this plugin to your Maven pom build section.

The ScalaTest options used in the examples below are added to the plugin’s excecution configuration according to.

<plugin>
   <groupId>org.scalatest</groupId>
   <artifactId>scalatest-maven-plugin</artifactId>
   <version>1.0-M2</version>
   <configuration>
      <reportsDirectory>${project.build.directory}/scalatest-reports</reportsDirectory>
      <junitxml>.</junitxml>
      <testFailureIgnore>true</testFailureIgnore>
      <filereports>WDF TestSuite.txt</filereports>
      <forkMode>never</forkMode>
      <parallel>${runSpecsInParallel}</parallel>
   </configuration>
   <executions>
      <execution>
         <id>test</id>
         <goals>
            <goal>test</goal>
         </goals>
         <configuration>
            <membersOnlySuites>${membersOnlySuites}</membersOnlySuites>
            <suites>${suites}</suites>
            <tagsToExclude>${tagsToExclude}</tagsToExclude>
            <tagsToInclude>${tagsToInclude}</tagsToInclude>
            <wildcardSuites>${wildcardSuites}</wildcardSuites>
         </configuration>
      </execution>
   </executions>
</plugin>
maven-profiles

For the majority of the use cases below we are using Maven project profiles the are piped to the Maven execution using the -P options. By example: mvn test -Pmyprofile1

The profiles we are setting up are defining NONE or more of the ScalaTest configuration options described above.

<profile>
   <id>myprofile1</id>
   <properties>
      <tagsToExclude>SlowTest</tagsToExclude>
      <membersOnlySuites>com.mycompany.app.api</membersOnlySuites>
   </properties>
</profile>
Jenkins jobs

These are straight forward, create a maven job that runs a goal in line with: mvn test -Pmyprofile1.

Use cases addressed

  1. Running a selected set of test specs based on a package structure using, membersOnlySuites and wildcardSuites
  2. Running a selected set if tests based on spec names using suites
  3. Running a selected set of tests that are tagged using tagsToInclude
  4. Running a selected set of tests but exluding tests that have a certain tag
  5. Running tests with a given profile and overriding scalatest properties

Assuming the following test spec structure for all examples below

|- com.mycompany.app.api.v1
| |- CreateEntitySpec.scala
| \- DeleteEntitySpec.scala
\- com.mycompany.app.api.v2

Selected set of tests based on package structure

The membersOnlySuites picks up specs that are directly placed in the given package. It does not pick up any specs in its sub-packages.

// Runs all v1 tests
mvn test -DmembersOnlySuites=com.mycompany.app.api.v1
// Runs all v1 and v2 tests
mvn test -DmembersOnlySuites=com.mycompany.app.api.v1,com.mycompany.app.api.v2

The property wildcardSuites on the other hand will pick up all specs in the given package and all sub packages.

// Runs all v1 and v2 tests
mvn test -DwildcardSuites=com.mycompany.app.api

 

Running a selected set if tests based on spec names

// Runs all tests in the CreateEntitySpec and DeleteEntitySpec
mvn test -Dsuites=com.mycompany.app.api.v1.CreateEntitySpec,com.mycompany.app.api.v1.DeleteEntitySpec

 

Running a selected set of tests that are tagged

// Runs all tests in the CreateEntitySpec and DeleteEntitySpec that are tagged as SmokeTest and/or FastTest
mvn test -Dsuites=com.mycompany.app.api.v1.CreateEntitySpec,com.mycompany.app.api.v1.DeleteEntitySpec
-DtagsToInclude=SmokeTest,FastTest

 

Running a selected set of tests but exluding tests that have a certain tag

// Runs all test cases in all test spec but not those tests that are tagged as VerySlowTest
mvn test -DmembersOnlySuites=com.mycompany.app.api -DtagsToExclude=VerySlowTest

 

Running tests with a given profile and overriding scalatest properties

Resetting properties are done using the value None otherwise just overwrite by giving it a new value.

// Runs all tests as given in the myprofile1 profile in the pom file resetting
// the tagsToExclude property set in the profile in the pom.
mvn test -Pmyprofile1 -DtagsToExclude=None
Advertisements

Bundling ChromeDriver with your test code

This post will just exemplify one way of maintaining the chrome drivers used for running automated tests with WebDriver and the Chrome browser without having to update and install the ChromeDriver to all possible nodes where tests will be running. The ChromeDriver will simply be bundled with the running tests and put under source control as for all other test ware.

Downloading links
http://code.google.com/p/selenium/wiki/ChromeDriver

Bundling
Start by putting the the drivers under the resource folder so that they will be picked up by Maven per default.

<project root>/src/main/resources/chromedriver/mac/chromedriver
<project root>/src/main/resources/chromedriver/windows/chromedriver.exe

Implementation
There are of course different drivers for different OS types and this needs to be handled using the os.name system property.

As per the ChromeDriver usage instructions (here) a system property has to be set pointing to the ChromeDriver server to use for the Chrome browser bridging. We will not point to a fixed location in the file system, instead well get the path by using the Class.getResource which will enable us to bundle ChromeDriver inside our test framework even if it is bundled into a jar file.

Basically what should be done are the following steps.

  • Determine OS type
  • Get the Chrome Driver resource and make sure it is executable using File.setExecutable(true). This is due to when packaged in a jar the execution attributes ‘x’ will be removed on Mac (and assumed on Linux too).
  • Set the “web driver.chrome.driver” system property.
  • Check that a Chrome installation exists in the default location [OPTIONAL]
private static WebDriver driver = null;
// The ChromeDriver locations under the resource folder
private static String MAC_DRIVER = "/chromedriver/mac/chromedriver";
private static String WINDOWS_DRIVER = "/chromedriver/windows/chromedriver.exe";

public static void setupChromeDriver() {
   // OS type
   if (System.getProperty("os.name").contains("Mac")) {
      File cDriver = new File(Tester.class.getResource(MAC_DRIVER).getFile());

      // Is it executable
      if (!cDriver.canExecute()) {
         cDriver.setExecutable(true);
      }
      System.setProperty("webdriver.chrome.driver", Tester.class.getResource(MAC_DRIVER).getFile());

      // Now checking for existence of Chrome executable.'
      if (!new File("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome").exists()) {
         throw new RuntimeException(errorMessage);
      }
   } else {
      System.setProperty("webdriver.chrome.driver", Tester.class.getResource(WINDOWS_DRIVER).getFile());

      // Now checking for existence of Chrome executable.'
      if (!new File(System.getProperty("user.home") + "/AppData/Local/Google/Chrome/Application/chrome.exe").exists()) {
         throw new RuntimeException(errorMessage);
      }
   }

   ChromeOptions options = new ChromeOptions();
   options.addArguments("--start-maximized");
   options.addArguments("--ignore-certificate-errors");
   driver = new ChromeDriver(options);
}

Test case example
Pretty straight on from here, setup WebDriver through the implemented method above and run a simple open page test to see that things worked out.

private static WebDriver driver = null;
public static void setupChromeDriver(){
   ...
}

@BeforeClass
public static void setupTestClass() throws Exception {
   setupChromeDriver();
}

@Test
public void demoTestCase() throws Exception {
   driver.get("http://code.google.com/p/selenium/wiki/ChromeDriver");
   Thread.sleep(1000);
}

Using JUnit @Category and Maven profiles to get stable test suites

Since it happens over and over again, no matter what project I am in. Keeping the test suites green and passing is a …

The blind eye

The one sneaking little test case that starts failing can prove to be a real killer to all automation approaches. More than once it has shown that as soon as a single test case breaks a test suite things starts to go downhill, fast. Ok, there could be acceptable resons for the test case to be broken, e.g. if there are issues in the system under test or in the test framework that are not bugs.

The usual decision taken is to leave the test case as is for now since we are all aware that it will fail for a while, BAD choice!!! All attention on the failing test suite is moving from slight attention to abandoned, and if other test in the test same test suite starts failing for various reasons it will most likely go by undetected.

What can/should be done here

  • Disable the test case and enable it again when it should be back on track
  • Maintain test suites to allow it to execute and fail but in a another suite/run to keep the MUST always pass test suites all green

The later case is the way to go.

Maven profiles to your rescue

Using Maven profiles it is easy to group your JUnit test classes and test cases in different categories. Usually this is used for grouping tests into slow, fast and browser-less tests a.s.f.. This approach can be stretched even further by using a grouping that reflects the current state of the test case. There does not have to be that many different groupable categories but at least two is needed.

  • Stable
  • Unstable/Maturing

Stable is self explained while Unstable/Maturing covers everything that is NOT always PASSING. It might be test cases that are unstable beacuse of;

  • timing issues
  • tested functionality is still under development (during sprints)
  • there is a low priority bug that will be fixed later
  • unknown issues causing it to sporadically fail
  • waiting to get a STABLE stamp

Use case 1
The last item is an approach that can be used to be really be sure that the stable test suites does not get polluted. Any new test case needs to run a set number of times (e.g. 20) in a test suite and must pass all all times before being stamped as a reliable test case.

Use case 2
When a bug surfaces that causes a test case to fail it might sometimes be the case that it wont be fixed for some time period. Instead of totally disabling or trying to fix the test case that would fail an otherwise stable test suite it shall be moved to an unstable test suite. This is a way to put the test case in a quarantine until the bug has been fixed and on the same tame make sure it still compiles and is executable.

Categorizing tests using JUnit @Category annotation
Use the JUnit @Category annotation in front of your test cases or test classes.

@Category(x.y.z.Stable.class)
@Test
public void aTestCase() {
  // void
}

// 
// Alternatively an unstable test case
//

@Category(x.y.z.Unstable.class)
@Test
public void anotherTestCase() {
  // void
}

Splitting test execution using Maven profiles

Configure your Surefire plugin to include and exclude certain test classes as well as determine a set of JUnit @Category groups to be included in the test run. Set up your pom including profiles as examplified below.

<build>	
	<plugins>	
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>2.12.2</version>
			<dependencies>
				<dependency>
					<groupId>org.apache.maven.surefire</groupId>
					<artifactId>surefire-junit47</artifactId>
					<version>2.12.2</version>
				</dependency>
			</dependencies>
			<configuration>
				<testFailureIgnore>true</testFailureIgnore>
				<excludes>
					<exclude>${exclude.tests}</exclude>
				</excludes>
				<includes>
					<include>${include.tests}</include>
				</includes>
				<groups>${testcase.groups}</groups>
				<configuration>
					<forkMode>never</forkMode>
					<runOrder>random</runOrder>
				</configuration>
			</configuration>
		</plugin>
	</plugins>
</build>

The testcase.groups property can be used to define a set of groups (JUnit @Categories) to include when tests using the actual profile.

<profiles>
	<profile>
		<id>stable-tests</id>
		<properties>
			<exclude.tests>**/x/**/*.java</exclude.tests>
			<include.tests>**/y/**/*.java</include.tests>
			<testcase.groups>x.y.z.Stable</testcase.groups>
		</properties>
	</profile>
	<profile>
		<id>unstable-tests</id>
		<properties>
			<exclude.tests>**/x/**/*.java</exclude.tests>
			<include.tests>**/y/**/*.java</include.tests>
			<testcase.groups>x.y.z.Unstable,x.y.z.Maturing</testcase.groups>
		</properties>
	</profile>
</profiles>

Packaging and referencing images for Sikuli based tests in artifacts

This post describes one approach for packaging/bundling image resources used for Sikuli based automated test cases.

  • Bundling images as resources
  • Retrieveing/using image resources

That the images used when automating tests using Sikuli needs to versioned should be rather obvious. Naturally the majority of the pictures used in test cases reflects part of the system under test and hance shall be treated and handled as a part of the test code synchronized with what we are actually testing against. This is usually solved by having the test code in the same repository as the code under test, though it might not always be the case.

In a current test setup it was not possible to keep the test code and images together with the code under test so the test code had to be distrubited as an versioned artifact/jar-file.

Test artifact

One if the easier ways for avoiding a lot of test case maintenance is to wrap the interface to the system under test in to an artifact/API/jar-file. This allows for the artifact to be shared between different source control repositories and included in automated tests residing elsewhere (usual scenario when there are multiple trunks and components in a system). For the artifact approach the images to use for the Sikuli based tests needs to be a part of this artifact.

Bundling the images
Out of the box really, well if you are using Maven. Just put the images under resources and they gets bundled correctly in the jar. The location in the repository should be something like this.


src\main
  \- java
      \- org.project.example
  \- resources\images
      \- theImageToLookFor.PNG
      \- otherImage.PNG
      \- ...

That is all there is to it, now distribute the artifact freely, a complete pom for building a complete Sikuli test artifact including images is posted at the end.

Accessing the images in the test code
To be able to reference and use the images in your test there needs to be functionality in the artifact for getting the image resource. So implement a generic getter method that picks up all images correctly for you from inside the jar/artifact during run-time.

class TestArtifact {
	// Image file name must start with the '/' and the image name is
	// case sensitive so be thorough when using.
	private Pattern getPattern(String fileName) throws IOException {
	   BufferedImage image = ImageIO.read(getClass().getResource(fileName));
	   return new Pattern(image);
	}

	// Example method using an image.
	// Screen screen = new Screen();
	public clickOnImage(String imageName) throws Exception {
		Match m = screen.find(getPattern(imageName));
		screen.click(m);
	}
}

Usage inside test cases
Note that when referencing images the path has to start with ‘/’ (fowardslash) to make sure the image is picked up correctly from within the artifact.

class Test {
	private static TestArtifact sikuli = new TestArtifact();
	
	@Test
	public void shallClickImage() throws Exception {
		sikuli.clickOnImage("/images/theImageToLookFor.PNG");
	}
}

Example of artifact pom

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.yourcompany.automation.apis</groupId>
    <artifactId>sikuliwrapper</artifactId>
    <packaging>jar</packaging>
    <name>Sikili wrapper test artifact</name>
    <version>1.0.0-SNAPSHOT</version>
    <inceptionYear>2012</inceptionYear>

    <dependencies>
        <dependency>
            <groupId>org.sikuli</groupId>
            <artifactId>sikuli-script</artifactId>
            <version>0.10.2</version>
            <type>jar</type>
        </dependency>
    </dependencies>

</project>