SonarQube quality gate feedback in a Slack channel

Well it is easy to agree on that Slack is awesome for team collaboration and with the hooks it has available it is rather easy to get some decent feedback into the team channels.

There is a Jenkins CI plugin available for Slack that pushes build notifications into a given channel. But that merely pushes what Jenkins provides, failed, unstable asf.. If you are working with SonarQube and want to know a bit more than if the build turned unstable, then there is an additional Jenkins plugin available for you, SonarQube-Slack-Pusher

… which would push a notification like this in a Slack channel.

sonar-slack-pusher-notification

What needs to be in place

  1. QualityGate for a SonarQube project
  2. Incoming WebHook integration to your Slack channel
  3. Installed Jenkins plugin, Sonar-Slack-Pusher
  4. Configured Jenkins job pushing to Slack

Quality gates
The notification above highlights a quality gate that has failed in SonarQube. A quality gate is a way of highlighting that we are not meeting expected quality criteria for a SonarQube project. In the SonarQube UI this is visualised by read and yellow highlighting and could be things like warnings in the code or not enough unit test coverage.

Well, yes a quality gate needs to be defined in SonarQube. It could be linked to an existing project already but it is also possible to pass the quality gate as a part of running a SonarQube analysis job.

Configure the Slack channel
The SonarQube project/’quality gate’ and the Slack channel is there. Add an Incoming WebHook to the channel and note down the hook URL, you will need it when configuring the Jenkins job, you will also need channel admin rights to perform this step.

Pushing quality gate information from Jenkins to the Slack channel
Assuming that there is a Jenkins job that triggers and runs a SonarQube pipeline, i.e. pushing the results to SonarQube. Add the Sonar-Slack-Pusher plugin to your Jenkins installation and add the corresponding Post-build Action to she SonarQube job.

Configure it to use

  • hook URL given when configuring the Slack WebHook
  • sonar URL
  • SonarQube job name and if using branches add that as well.

jenkins-config

Bam all set! Run the job and if you have any failing quality gates you will get a notification in your channel.

Overview of sporadically failing test cases in Jenkins – UnitTH Jenkins plugin

Never, ever a stable blue build right? It does not seem to matter what you do, but for system integration tests there always seems to one or a few sporadically failing test cases. What can be extra annoying is that there might always be different test cases that sporadically fail, not the same one or two. To be able to determine the success of executed test suites there is a need to get an overview of the last few runs. The easiest way to visualise this is to create a matrix of test runs versus test cases to get a hit map with frequency of failures for specific test cases.

unitth-matrix

Test case failure spread.

For those using Jenkins there is a small Jenkins plugin at Sourceforge that can be added to the post build steps of any job that generates test results. The plugin gives you useful overview and stats for the available build test results.

Installation

Usage

  • Edit the configuration of a Jenkins job and in the post build section select the plugin
  • Run the job again to get the first matrix report, there will be a link in the sidebar
  • Every failed run of a test case has a link to the report trace in the build it was executed, simply click on the red hits in the matrix

ScalaTests on Jenkins using Maven, use case examples

This is a post for how to do it with Maven, the same functionality can be acheived by using Sbt if preferring that option.

There are a number of threads out there that addresses the topic of configuring and running ScalaTest test cases/specs on Jenkins, some threads older than others but since this seems to be an re-occuring question topic why not to examplify how it can be done based for some of the most common use cases that seems to be popping up in the forums. So if you have problems setting up jobs that are running specific test specs or test that are tagged then have a look at the examples below.

Core knowledge basics

scalatest-maven-plugin

This plugins enbles execution of ScalaTests tests using Maven without any extra fuzz like using @RunWith(classOf[JUnitRunner])… I takes a set of configuration options that are well defined on the . So start by adding this plugin to your Maven pom build section.

The ScalaTest options used in the examples below are added to the plugin’s excecution configuration according to.

<plugin>
   <groupId>org.scalatest</groupId>
   <artifactId>scalatest-maven-plugin</artifactId>
   <version>1.0-M2</version>
   <configuration>
      <reportsDirectory>${project.build.directory}/scalatest-reports</reportsDirectory>
      <junitxml>.</junitxml>
      <testFailureIgnore>true</testFailureIgnore>
      <filereports>WDF TestSuite.txt</filereports>
      <forkMode>never</forkMode>
      <parallel>${runSpecsInParallel}</parallel>
   </configuration>
   <executions>
      <execution>
         <id>test</id>
         <goals>
            <goal>test</goal>
         </goals>
         <configuration>
            <membersOnlySuites>${membersOnlySuites}</membersOnlySuites>
            <suites>${suites}</suites>
            <tagsToExclude>${tagsToExclude}</tagsToExclude>
            <tagsToInclude>${tagsToInclude}</tagsToInclude>
            <wildcardSuites>${wildcardSuites}</wildcardSuites>
         </configuration>
      </execution>
   </executions>
</plugin>
maven-profiles

For the majority of the use cases below we are using Maven project profiles the are piped to the Maven execution using the -P options. By example: mvn test -Pmyprofile1

The profiles we are setting up are defining NONE or more of the ScalaTest configuration options described above.

<profile>
   <id>myprofile1</id>
   <properties>
      <tagsToExclude>SlowTest</tagsToExclude>
      <membersOnlySuites>com.mycompany.app.api</membersOnlySuites>
   </properties>
</profile>
Jenkins jobs

These are straight forward, create a maven job that runs a goal in line with: mvn test -Pmyprofile1.

Use cases addressed

  1. Running a selected set of test specs based on a package structure using, membersOnlySuites and wildcardSuites
  2. Running a selected set if tests based on spec names using suites
  3. Running a selected set of tests that are tagged using tagsToInclude
  4. Running a selected set of tests but exluding tests that have a certain tag
  5. Running tests with a given profile and overriding scalatest properties

Assuming the following test spec structure for all examples below

|- com.mycompany.app.api.v1
| |- CreateEntitySpec.scala
| \- DeleteEntitySpec.scala
\- com.mycompany.app.api.v2

Selected set of tests based on package structure

The membersOnlySuites picks up specs that are directly placed in the given package. It does not pick up any specs in its sub-packages.

// Runs all v1 tests
mvn test -DmembersOnlySuites=com.mycompany.app.api.v1
// Runs all v1 and v2 tests
mvn test -DmembersOnlySuites=com.mycompany.app.api.v1,com.mycompany.app.api.v2

The property wildcardSuites on the other hand will pick up all specs in the given package and all sub packages.

// Runs all v1 and v2 tests
mvn test -DwildcardSuites=com.mycompany.app.api

 

Running a selected set if tests based on spec names

// Runs all tests in the CreateEntitySpec and DeleteEntitySpec
mvn test -Dsuites=com.mycompany.app.api.v1.CreateEntitySpec,com.mycompany.app.api.v1.DeleteEntitySpec

 

Running a selected set of tests that are tagged

// Runs all tests in the CreateEntitySpec and DeleteEntitySpec that are tagged as SmokeTest and/or FastTest
mvn test -Dsuites=com.mycompany.app.api.v1.CreateEntitySpec,com.mycompany.app.api.v1.DeleteEntitySpec
-DtagsToInclude=SmokeTest,FastTest

 

Running a selected set of tests but exluding tests that have a certain tag

// Runs all test cases in all test spec but not those tests that are tagged as VerySlowTest
mvn test -DmembersOnlySuites=com.mycompany.app.api -DtagsToExclude=VerySlowTest

 

Running tests with a given profile and overriding scalatest properties

Resetting properties are done using the value None otherwise just overwrite by giving it a new value.

// Runs all tests as given in the myprofile1 profile in the pom file resetting
// the tagsToExclude property set in the profile in the pom.
mvn test -Pmyprofile1 -DtagsToExclude=None

Setting up a Jenkins slave for Sikuli based tests

To be able to use the full power of Sikuli based test cases they have to work in the ‘bigger’ picture, running countinuosly in a Jenking master/slave setup. This post will go through the steps we needed to take to get Jenkins slaves (running Windows 7) to execute Sikuli based tests reliably in a larger Jenkins setup.

Covered …

  • Creating the local user
  • Disabling screen savers
  • Disabling screen lock
  • Hooking up the slave to the master

Basic requirements for a Sikuli test execution slave

To be able to use a Windows 7 based Jenkins slave for running Sikuli based tests there are a few things that needs to be understood first.

As soon as the the current user on a machine gets logged out or gets its screen locked any Sikuli based tests will stop working. Secondly to be able to run tests triggered through Jenkins the executing user (the user that runs the Jenkins process) needs to have access to the desktop to be able to run applications like browsers in a visible way. Running Jenkins as a web service in Windows 7 will prohibit visible execution of applications or in other words the service is not allowed to access the desktop. Meaning that things are not as simple as starting Jenkins as a Windows service and then hook it up to the Jenkins master. Java web start (JNLP) is the way to go.

Requirements in short …

  • local user that is always logged on
  • machine that never gets locked
  • node that connects to the Jenkins master using Java web start (JNLP)
  • Java and environment variables configured properly to run Sikuli-based test cases (here)

Getting the local user in place

1) Create a local superuser account on the slave node.

2) Set the following keys in the registry to ensure that the user never gets logged out.

Open the registry editor (regedit) and add a new DWORD as named DisableLockWorkstation and give it the value 1 (1 = disable) to the path below.

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System

3) It also has to be made sure that the local user always gets logged on automatically if the physical machine gets restarted. Open the following entry in the registry editor.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon

Make sure the following keys are present and that they are set accordingly.

AutoAdminLogon Value : 1
DefaultPassword Value : 'the password set for the created local super user'
DefaultUsername Value : 'the super user name'

Reboot to try it out.

Disabling all screen savers

One again using the registry, set the ScreenSaveActive key to 0 in the location below.

HKEY_CURRENT_USER\Control Panel\Desktop

Connecting to Jenkins master

1) On the test execution slave, open any browser that has java support
2) Navigate to the URL of the Jenkins master
3) Manage Jenkins -> Manage Nodes
4) Add a new node
5) Select the new node and click on the Launch button to connect to the master using JNLP, if the node already has all the correct Java installations and environment variables we are set to go (here).

JNLP connection launch button as seen in Jenkins

Troubleshooting

Firewalls, getting the JNLP connection to work might need changes in firewalls. Jenkins is by default configured to use a randomly generated port for the JNLP connection. Change this setting to use a fixed port number then there should not be to big of a trouble getting the firewall rules in place.

Manage Jenkins -> Configure System