“When is test automation ready?”, we hear from many clients. We sometimes find the question puzzling, almost a little philosophical. As long as you keep developing new features, you need to continue developing the test automation too, right?
But we think there is a deeper, underlying question lurking behind it:
“Who should be responsible for test automation?”
And even more in-depth:
“What are the different parts of test automation that one can be responsible for?”
The two parts of test automation: Harness & End User Workflows
We at Eficode like to divide test automation into two distinct parts:
Test Harness
A Test Harness can be defined in many ways, but we define it as “everything you need for a test case to execute”. The main components are:
- the chosen test tools
- the environments
- the test data
- the Continuous Delivery solution
- other supporting tools (e.g. Docker)
End User Workflows
This second part of test automation is about the content of a test case. What does it do and why? This is deeply linked with the requirements of the software itself, and the business problems it solves.
Have you ever wondered why there is so much “passing the buck” when trying to solve test failures? You hear things like “our test cases have failed in the pipeline, who could fix these?”. Well, this is the reason. The test might fail because there is something wrong in the Test Harness, or it might fail because the business content of the test case is not aligned with the application-under-test itself.
Ownership problems during the project
When testing can’t keep pace with feature development
In the early phases of the project, there is sort of an on-going race between feature development and Test Harness development. To test what has been developed, you also need to add new capabilities to the Test Harness. You may be tempted to take the easy way out with the Test Harness and “just get it done”.
But as the feature set grows and new test cases are automated, in our experience most of the time goes to analyzing why the existing test cases failed. This means you will have less and less time to write new test cases. Sound familiar?
When nobody takes responsibility
You will have problems if no-one takes responsibility for the End User Workflow part of test automation. In the worst case we’ve seen, the QA professionals can’t or won’t even speak with the feature developers – “I did not write these tests so I’m not responsible for their failures”.
These situations will hinder the test case analysis even more, potentially getting the team in serious trouble since the new features might not get tested (at least with automation).
Solving the ownership problems
There are many ways to tackle these two problems. For example:
- a dedicated QA professional or SDET position in a team can answer both challenges with the help of the rest of the team.
- a tester could be in charge of the End User Workflows while the development team is in charge of the Test Harness. This works wonderfully as it promotes cross-functional teams naturally.
- maybe the product owner or another business-responsible, with the support of a QA expert and the rest of the development team, are able to collaborate using ATDD – business writing the End User Workflows, with the development team automating it.
- one of the best ways to shift-left testing issues is to have the expert(s) present already when defining the new features, alongside other stakeholders, like security representatives. This way, testing and improving the Test Harness is part of the feature design.
There are no one-size-fits all solutions. It has to make sense within your unique context. But you do need to agree on the following while discussing the ways of working:
- How do we make a high-quality Test Harness?
- How do we make sure that End User Workflows are representative of what we actually want?
Test automation ownership during the maintenance phase
If your test cases constantly fail needlessly, if it takes forever to analyze test failures, or if it takes too long to even implement a test case, chances are you suffer from serious technical debt.
Code quality has evolved from the classical swear words per minute to how easy and frictionless it is to add and maintain automated test cases with your applications. If there is constant irritation, that’s a sign that it’s time to pay down the debt. It will serve you in the long run, since 60% of your software’s lifetime is spent in maintenance.
When your software project is in the maintenance phase, you shouldn’t expect too many changes anymore. And as with the software itself, the Test Harness and End User Workflows only need minimal updates.
But although it may not take a lot of work, it might make sense to move this work, alongside with the project, over to a team responsible for maintaining several applications. Often it doesn’t even make sense to do this in-house. It may make more sense to outsource both the application and the test automation maintenance.
In short: who should own test automation?
So who should own the test cases? We at Eficode think that everyone who is part of project development should own the tests. How the division of work between Test Harness and End User Workflows is done should be explicitly agreed upon between all parties.
In the end, tests are part of the product just like the source code is.
Published: Sep 1, 2022
