Third Horseman of your Test Automation Apocalypse - Noise
Expensive employees burn long hours building out automation infrastructure. Here is the third sign that investment will not be valued.
In the last part of this series, we looked at the needs of the business stakeholders of test automation. This time, we will look at the technical stakeholders. We are talking about the people who are looking at the automation daily: the test team, the application developers, the scrum masters and the dev managers.
As we said before, trust is the only currency in test automation, but what are these people trusting their automation for? A primary need test automation fills for the technical team is to provide a lane departure system for their development efforts.
When driving a car, a lane departure system (or rumble strips on the road) alerts us when a problem causes us to get dangerously off course. When developing software, if something is done the breaks that existing system, the technical folks wants to know fast, loud, and specifically what broke. If they find out immediately, that makes it fast and cheap to just stop and fix it. The longer it takes for them to discover that it is broken, the harder to find, the bigger impact the fix, and the more likely it is to have an impact on schedules or deliverables. That feedback loop is near the core of Agile practice. To stick with the steering metaphor, if the lane departure system has a delay in alerting, you may be off the road or in oncoming traffic by the time it goes off and it may be too late to head off big problems. Slow feedback does not break trust, though. No, the real problem is when the technical people stop listening to the alarm.
Technical people stop listening to the failures when they are sorting through any noise of false failures. False failures is specifically any failures that do not signal an application problem. This point reflects the flip side of the coin we discussed before. If the business is looking for meaningful green bars, the technical people are looking for meaningful red ones. A failure in the automation should be a call to action, for the team to immediately move to fix. Frequent false alarms quickly become background noise. Facing frequent alarms, the team delays in addressing those alarms right before they start ignoring them altogether. Common failures that are anything other than product failures cause test automation to be ignored so fast, I am sure it is a phenomenon worth a study in programmer psychology.
Regardless of why it is true, though, all sources of false failures are a danger to the health of your test automation project because they wear down the team’s trust in the automation and, therefore, their willingness to answer their call to action. Sources of those false positives can be in environments, deployments, test definitions, or in the automation code. Tackling all those issues reliably resembles, in many ways, the same challenges that organizations face in implementing their devops and continuous delivery pipelines. This may even be a place to develop and prove out those processes.
Avoiding false alarms in the test automation application itself requires building a test application that has a very high standard of reliability. That quickly ties back into the conclusions from part 1 of this series. Outstanding development practices are as important in “mere test code” as they are in the application under test. The teams building this code need great support and execution to be successful.
The goal is to eliminate all causes of test automation failure that are not directly driven by application errors. The noise of those false failures is the third alarm that the automation project is in danger.