Overview of sporadically failing test cases in Jenkins – UnitTH Jenkins plugin

Never, ever a stable blue build right? It does not seem to matter what you do, but for system integration tests there always seems to one or a few sporadically failing test cases. What can be extra annoying is that there might always be different test cases that sporadically fail, not the same one or two. To be able to determine the success of executed test suites there is a need to get an overview of the last few runs. The easiest way to visualise this is to create a matrix of test runs versus test cases to get a hit map with frequency of failures for specific test cases.

unitth-matrix

Test case failure spread.

For those using Jenkins there is a small Jenkins plugin at Sourceforge that can be added to the post build steps of any job that generates test results. The plugin gives you useful overview and stats for the available build test results.

Installation

Usage

  • Edit the configuration of a Jenkins job and in the post build section select the plugin
  • Run the job again to get the first matrix report, there will be a link in the sidebar
  • Every failed run of a test case has a link to the report trace in the build it was executed, simply click on the red hits in the matrix