Oxygen is a new tool for Robot Framework to consolidate all test reporting for greater visibility and better quality metrics.
Oxygen is a new open source tool created by Eficode to consolidate all separate test reports into one. Oxygen provides different ways to combine reports using Robot Framework’s reporting format and extensibility. This gives greater visibility and enables further analysis for quality metrics. Let me show you how it’s done.
From meaningless data to relevant information
Using multiple tools to test software is a great way to increase quality and therefore feel more confident about what you’re doing. However, each tool comes with its own kind of result report, meaning we need to look in multiple places to understand the big picture. They are structured differently, meaning you need to context switch which is a productivity sink. But worst of all, it requires developers to perform fuzzy interpretations to consolidate the results into a release decision.
Oxygen is a tool that combines distinct test reports using Robot Framework, the open source general automation framework. The consolidated results lead to better quality metrics and saves developers’ time. It can easily be integrated into your workflow and even into automated pipelines.
Choosing the right tool for the job
One of the fundamental tenets of DevOps is choosing the right tool for the job. In the world of test automation, some of the most common tools include:
- A low-level technical testing tool to demonstrate that the software works as the programmer intended
- A high-level end-to-end testing tool to demonstrate that the business requirements are fulfilled and functionally work
- A performance testing tool to give us confidence that software is efficient enough
- A security testing tool to check that we do not have obvious security concerns.
Depending on business context, the list of specialized testing tools a project might need is endless. Our customers often need, in addition to the above, tools to test the following:
- Critical API integrations
- Infrastructure when using Infrastructure-as-Code (IaC)
- Accessibility standards like WCAG
- Regulatory traceability in heavy industries
Making sense of it all
Interpreting the different reports produced by these tools creates knowledge silos within a project. Only a few individuals can surmise what the true state of the project is, leading to the infamous bus factor.
All of the data from these reports is commonly distilled by displaying the data as metrics on a dashboard. In reality, this leads to either extensive maintenance efforts to keep the metrics running, or limiting the choice of tools based on what has already been integrated with the dashboard solution.
Maintenance efforts are hard to justify to management as resources are scarce as it is. Likewise, limiting the tool options does not empower teams to use the tools they need which can lead to inefficient and meaningless work.
If a small set of testing tools are poorly chosen, it might even compromise how accurate our quality metrics are. For example, some widely used unit testing tools do not give the developer the ability to denote the purpose of a test case. This lumps all test cases together as unit tests, even though some of them might be component, integration, or system tests.
Introducing Oxygen for Robot Framework
Eficode is participating in the European-wide research program Testomat Project funded by Business Finland. Based on its findings, Eficode has created a new tool for Robot Framework which provides a straightforward way to consolidate different test reports into one.
When using Oxygen, you can write acceptance tests corresponding to each tool you want to run. Then, Oxygen automatically transforms their reports into Robot Framework test cases, thus integrating everything together. For each test case the other tool executed, there will be a corresponding test case in Robot Framework’s log and report with relevant information intact, such as the test case’s status.
Let’s check out what they look like. Below we have four Robot Framework test cases. The first two run JUnit test cases via Maven, the third runs performance tests with Gatling, and the last one runs a security scanner with ZAP. As you can see from the first two, we can separate test executions made with a single tool. That way you can give more context on their purpose.
*** Test cases ***
Java unit tests should pass
Prepare environment platform=${PLATFORM}
Run JUnit path/to/results.xml mvn clean test
Java integration tests should pass
[Tags] integration-tests
Prepare environment platform=${PLATFORM}
Run JUnit another/path/results.xml mvn clean integration-test
Performance should not degrade
${gatling output folder}= Set variable path/to/simulations
Run Gatling ${gatling output folder}/gatling.log
... %{GATLING_HOME}/bin/gatling.sh --results-folder ${gatling output folder} --simulation MyStressTest
Application should not have security holes
Run ZAP path/to/zap.json node my_zap_active_scan.js
Example from Oxygen’s keyword documentation
If you need to run different test tools in separate CI/CD pipelines or in different environments, Oxygen can provide a command line interface that integrates easily into any CI/CD pipeline. You can turn a single test report provided by another tool into a single Robot Framework’s output.xml like this:
$ python -m oxygen oxygen.junit my_junit_results.xml
You can then combine the resulting file with other test results with Robot Framework’s built-in rebot tool.
$ rebot my_junit_results_robot_output.xml my_other_test_results.xml output.xml
Extensible tool for diverse needs
At the time of writing, Oxygen has built-in support for three test tools: JUnit, Gatling, and ZAP. However, a key design goal of Oxygen is that it should be extendable by anyone. By harnessing the spirit of open source, developers can integrate other test reporting formats – that way everyone benefits.
Oxygen extends the functionality of Robot Framework and in the same way you can extend it as well to meet your reporting needs. Oxygen can be extended by writing a handler: a simple Python class that provides 1) a Robot Framework keyword to run the other tool and 2) a method that parses the report and returns what information it wants as a simple nested dictionary.
An example handler below provides a Robot Framework keyword Run My Tests which writes test results in a file in JSON format. The method parse_results() reads this file and creates a dictionary representation of the results that Oxygen understands.
import json
from oxygen import BaseHandler
class MyParser(BaseHandler):
'''
MyParser is an example how to extend Oxygen by writing your own handler
'''
RESULTS = {'tests': [
{'name': 'My test 1', 'passed': True, 'msg': ''},
{'name': 'My Test 2', 'passed': False, 'msg': 'Error text D:'}]
}
def run_my_tests(self, resultfile):
with open(resultfile, 'w') as f:
json.dump(self.RESULTS, f)
return resultfile
def parse_results(self, resultfile):
with open(resultfile, 'r') as f:
results = json.load(f)
return {
'name': resultfile,
'tags': [],
'setup': None,
'teardown': None,
'suites': [],
'tests': [{
'name': test['name'],
'tags': [],
'setup': None,
'teardown': None,
'keywords': [{
'name': test['name'] + ' result',
'pass': test['passed'],
'elapsed': 0.0,
'tags': [],
'messages': [test['msg']],
'teardown': None,
'keywords': []
}]
} for test in results['tests']]
}
Example handler from Oxygen repository.
The example presented above is a toy example intended to illustrate the two core concepts. The first is a method that OxygenLibrary provides as a keyword that runs the external test tool and the second is a method that transforms what you want from the external test tool report into Oxygen’s representation. From this point onwards, Oxygen knows how everything is put into Robot Framework log and report. It even provides an extendable or overridable command line interface for your handler automatically.
Eficode is in the process of writing an in-depth tutorial, so stay tuned!
Oxygen enables better quality metrics
Once you have consolidated your test reports into a unified Robot Framework report, you can easily ingest the data into a database with tools like InfluxDB Plugin or DbBot. You can then use this to generate visualizations with tools like Grafana. From there, analyzing the data and creating expressive metrics is a breeze.
Oxygen operates on Robot Framework test cases so test case tags added to the original test case become part of the results as well. This means you can categorize test cases regardless of whether the external test tool has such a feature (I’m looking at you, XCTest). You can then, for example, separate unit tests from integration tests. Since tags are a powerful feature in Robot Framework, they can be used with Oxygen too: you can collect statistics, separate critical test cases from lesser test cases, define test executions, and select what to run at which stage of the CI/CD pipeline.
Feel free to head on over to the Oxygen issue tracker to suggest new features or report bugs. If you build an extension for Oxygen, we hope you publish it in the Python Package Index and let us know, so we can link back to you.
Published: Jul 11, 2020
Updated: Sep 30, 2024
