CICD — what to do about failing tests

Published in

Time Travel Debugging

5 min readJun 26, 2020

The explosion of Agile development and DevOps practices has resulted in more frequent testing to enable faster and continuous delivery of software applications. That’s the theory anyway.

A study carried out recently by a Cambridge Judge Business School MBA project reveals that while the adoption of continuous integration is on the rise (growing from 70% in 2015 to 88% in 2019), software failures in QA & test still detrimentally affect delivery speed. This is the Great Stink in software development.

In this post, we’re going to talk about what development teams can do to make the problem of failing tests go away.

Failing tests are like ticking timebombs

The only way to unlock the true benefits of continuous integration is to achieve a good level of continuous delivery (CD). Achieving this requires having a zero-tolerance of failing tests. Tests must pass reliably and when they don’t, to avoid delivery delays, failing test results need to be actionable and so that they can be resolved quickly. The other option is to deploy software with unresolved defects — if that’s the case, then it’s not a matter of if, but when, they will blow up in everyone’s faces.

Ignore failing tests at your peril

Ignoring failing tests results can result in the following nightmare outcomes:

The effectiveness of testing is severely undermined by ignoring the test results. By definition, if undertaking testing, you are attempting to evaluate something (i.e. quality/ security/ reliability/ performance etc). There is absolutely no point in checking and testing for something if you’re going to ignore the results.
When put aside, failing tests are tolerated (with a promise to ourselves to fix it later) and often forgotten about. Before you know it, you have an ever growing , unmanageable backlog of unresolved failing tests.
Technical debt is increased as is the risk of unexploded timebomb bugs going off in production. They result in substantial costs to the business such as disruption of customer service, loss of revenue and the reputation loss from the perception of being a provider of unreliable software services.

LiveRecorder
Leading software failure replay platform

Software failure replay enables developers to record a failing process, down to instruction level, in both development or production. Traditional methods of debugging rely on being able to reproduce failures and guesswork when trying to figure out the root cause.

LiveRecorder is the leading software failure replay platform because it simply records what actually happened. The recording represents a carbon copy of what occurred — no reproducibility time needed.

The recording can then be replayed and debugged on any machine. Using LiveRecorder’s reverse debugging features, developers can quickly zero-in on the problem bug, deploy a fix once and resolve the problem.

Resolve failing faster tests with LiveRecorder

Implementing deployment pipelines using Continuous Integration tools is now commonplace and the most efficient way to manage the tasks associated with building, testing and deploying software.

A simple pipeline in Jenkins, a popular CI/CD tool, looks something like this.

* Jenkins pipeline, using the Blue Ocean user interface

If all goes well and tests turn green, the software release is fit for deployment. But if you have failures, they need to be dealt with and resolved.

Using LiveRecorder, an additional enhancement step can be added to this workflow. An extra stage can be added to this automated workflow so that failing tests are rerun using LiveRecorder.

Test failures are captured as a recording and stored as Jenkins artifacts. The recording captures the software failure, and provides all of the actionable failure insight data a developer needs to investigate the problem.

The recording contains an automatic reproduction of the failure, so no time needs to be spent on trying to reproduce the test failure and what caused it.

LiveRecorder recordings are portable and can be debugged by a developer on any machine.

Developers can then debug the recording by replaying the recording artifact forwards and backwards — getting full visibility and data-driven insight into what their software really did before it failed or misbehaved.

As a result, root cause detection time is significantly reduced. And developers can get straight to debugging the recording artefact — reducing the number of loops in agile development cycles and therefore increasing development velocity.

This video takes you through two demos that show exactly how LiveRecorder improves the efficiency of your CI/CD pipeline by providing a solution for rapidly resolving failing tests.

Bottom line

So, the bottom line is: where most testing methods and application performance monitoring tools are good at alerting developers to code problems and test failures; their effectiveness is limited to providing indicators and signals rather than quickly actionable data.

As a result, the efficiency of tests significantly drops. These tools can help to uncover code problems but a developer then has to investigate and track down the root cause with very few clues to go on. They fall short of giving developers the exact information they need to fix the problem rapidly in order to maintain release momentum.

By integrating LiveRecorder into your CI pipeline to record failing tests, the recordings provide developers with everything they need to understand the errors, debug and fix them.

By accelerating the MTTR of errors that cause failing tests, software failure replay platforms such as LiveRecorder, enable developers to resolve the defects that cause failing tests faster and therefore to achieve continuous delivery at speed.