Never, ever a stable blue build right? It does not seem to matter what you do, but for system integration tests there always seems to one or a few sporadically failing test cases. What can be extra annoying is that there might always be different test cases that sporadically fail, not the same one or two. To be able to determine the success of executed test suites there is a need to get an overview of the last few runs. The easiest way to visualise this is to create a matrix of test runs versus test cases to get a hit map with frequency of failures for specific test cases.
For those using Jenkins there is a small Jenkins plugin at Sourceforge that can be added to the post build steps of any job that generates test results. The plugin gives you useful overview and stats for the available build test results.
- Download the plugin from https://github.com/andnyb/unitth/releases
- In Jenkins navigate to manage plugins / advanced and select upload plugin
- Edit the configuration of a Jenkins job and in the post build section select the plugin
- Run the job again to get the first matrix report, there will be a link in the sidebar
- Every failed run of a test case has a link to the report trace in the build it was executed, simply click on the red hits in the matrix