Improving test coverage and performance with Test Pyramid

Writing automated tests can be a challenging task due to the particularity of this kind of coding. If your plan is to use them inside a Devops software development process,  which uses intensive automation, both for testing and infrastructure as a code, it’s important to make tests run fast and reliably. Unreliable tests have their own definition, as described at Google Testing Blog, “Flaky tests are tests that exhibit both a passing and a failing result with the same code”. Those types of tests can ruin a pipeline, causing false positives and leading the QA team to draw an incorrect conclusion of the test suite pass rate.

UI tests provide an approach that follows the user behavior, so you will assert the functionality of the entire software from the users perspective, providing a concrete business value, as opposed to testing if your function that generates a JSON is in the correct format, for example. This is the main point of the Google Testing Blog article, “Just Say No to More End-to-End Tests”, which discusses the practical aspects of UI testing, describing them as flaky, hard to point to the root cause and, hiding small bugs inside big bugs as most exceptions are suppressed and only custom warnings are exhibited in the UI.

To facilitate a change in this paradigm, Martin Fowler presents the Test Pyramid, a great visual metaphor directing you to think about the different layers of testing, and how many tests to distribute relatively in each layer. This pyramid was first mentioned in Mike Cohn’s book, “Succeeding with Agile” however, Martin comments that this is overly simplistic and can, therefore, be misleading.

Let’s see the modern concept of the Test Pyramid, updated by Martin:

The pyramid’s shape is useful to demonstrate two points about this approach: firstly, the right amount of tests to be implemented in each layer, so in the bottom, there are more tests compared to the top level. Secondly, it’s possible to expose the resource usage to run each layer, starting with unit tests in the bottom, that run quickly with no third party apps, compared to UI tests that are in the top and use a browser and Selenium Server (example). The distribution of the categories are completely aligned with these two metrics, let’s elaborate:

1. Unit tests: Focused on testing the smallest function in the code.
   The goal is to verify that the function knows how to receive the input and the processing that was designed. You can mock or stub the input to make it independent of other components.
2. Component Tests: Test an entire component of the software, without the UI.
   The component is a module of the application. At this level, the test starts to assert business rules, since all minor functions were tested at unit tests layer.
3. Integration Tests: Testing communication between two components in the architecture.
   The goal here is to verify if two components can talk to each other. If these two components use information from other parts of the system, such as APIs or other modules, they can be mocked or stubbed. Some other tests can be included at this level:
   1. Contract Test, which verifies if some API is following its pattern.
   2. Consumer Test, which verifies if some part of the code that receives information from other APIs, can consume it. The input can be mocked too.
4. End-to-End: Test the whole software throughout the UI.
5. Exploratory/Manual Tests: Some UI tests are so complex that it may be difficult to automate, so they can be run manually. Also, there are some complex flows that can be captured by a tool that analyzes application logs, like Splunk, so the QA team can map this and take action to automate a test.

Unit tests are easy to write and run at very low cost. When writing them, time should be taken to assert functionality and narrow down cases of whether: the function works with the anticipated/expected input, and how it deals with unexpected/invalid inputs. At the component test layer, it’s time to forget small functionalities covered on unit tests and assert business rules in the software.

For integration tests, the coverage must be further than the expected cases in the components communication and be capable of understanding a components error messages and when to return exceptions that can be treated by the application. Don’t forget to stub or mock what is not being tested in this scope. If you are testing database communication, you don’t need to reach the real third-party APIs, for example.

The End-to-End must test the main user flows into the application. These tests must be fewer in quantity than previous tests since they already cover a large part of the application; it’s not necessary to duplicate tests.

As a closing note:

Testing applications using only UI tests have huge long term costs and cause uncertainty due to their flakiness. The Test Pyramid is a good practice for your Devops Software Development process because it costs less, and is fast and reliable.