CICD — what to do about failing tests

The explosion of Agile development and DevOps practices has resulted in more frequent testing to enable faster and continuous delivery of software applications. That’s the theory anyway.

A study carried out recently by a Cambridge Judge Business School MBA project reveals that while the adoption of continuous integration is on the rise (growing from 70% in 2015 to 88% in 2019), software failures in QA & test still detrimentally affect delivery speed. This is the Great Stink in software development.

In this post, we’re going to talk about what development teams can do to make the problem of failing tests go away.

Failing tests are like ticking timebombs

Ignore failing tests at your peril

  1. The effectiveness of testing is severely undermined by ignoring the test results. By definition, if undertaking testing, you are attempting to evaluate something (i.e. quality/ security/ reliability/ performance etc). There is absolutely no point in checking and testing for something if you’re going to ignore the results.
  2. When put aside, failing tests are tolerated (with a promise to ourselves to fix it later) and often forgotten about. Before you know it, you have an ever growing , unmanageable backlog of unresolved failing tests.
  3. Technical debt is increased as is the risk of unexploded timebomb bugs going off in production. They result in substantial costs to the business such as disruption of customer service, loss of revenue and the reputation loss from the perception of being a provider of unreliable software services.

Leading software failure replay platform

LiveRecorder is the leading software failure replay platform because it simply records what actually happened. The recording represents a carbon copy of what occurred — no reproducibility time needed.

The recording can then be replayed and debugged on any machine. Using LiveRecorder’s reverse debugging features, developers can quickly zero-in on the problem bug, deploy a fix once and resolve the problem.

Resolve failing faster tests with LiveRecorder

A simple pipeline in Jenkins, a popular CI/CD tool, looks something like this.

* Jenkins pipeline, using the Blue Ocean user interface

If all goes well and tests turn green, the software release is fit for deployment. But if you have failures, they need to be dealt with and resolved.

Using LiveRecorder, an additional enhancement step can be added to this workflow. An extra stage can be added to this automated workflow so that failing tests are rerun using LiveRecorder.

Test failures are captured as a recording and stored as Jenkins artifacts. The recording captures the software failure, and provides all of the actionable failure insight data a developer needs to investigate the problem.

The recording contains an automatic reproduction of the failure, so no time needs to be spent on trying to reproduce the test failure and what caused it.

LiveRecorder recordings are portable and can be debugged by a developer on any machine.

Developers can then debug the recording by replaying the recording artifact forwards and backwards — getting full visibility and data-driven insight into what their software really did before it failed or misbehaved.

As a result, root cause detection time is significantly reduced. And developers can get straight to debugging the recording artefact — reducing the number of loops in agile development cycles and therefore increasing development velocity.

This video takes you through two demos that show exactly how LiveRecorder improves the efficiency of your CI/CD pipeline by providing a solution for rapidly resolving failing tests.

Bottom line

As a result, the efficiency of tests significantly drops. These tools can help to uncover code problems but a developer then has to investigate and track down the root cause with very few clues to go on. They fall short of giving developers the exact information they need to fix the problem rapidly in order to maintain release momentum.

By integrating LiveRecorder into your CI pipeline to record failing tests, the recordings provide developers with everything they need to understand the errors, debug and fix them.

By accelerating the MTTR of errors that cause failing tests, software failure replay platforms such as LiveRecorder, enable developers to resolve the defects that cause failing tests faster and therefore to achieve continuous delivery at speed.

Zoe Laycock head of Product Marketing at
Zoe Laycock head of Product Marketing at

Zoe Laycock
Head of Product Marketing

Originally published at

LiveRecorder is an enterprise software failure replay platform. We help developers Record, Replay & Resolve bugs & accelerate MTTR.