The ultimate guide: Test management for Atlassian Jira

It is not unheard of to manage your test cases in Jira. Issues are a great container to hold test specifications and steps within it.

Traditionally, if you wanted to do test management using Jira, you would have to create new custom fields with test steps and a separate field for requirements. This often feels clumsy and makes people drift away from Jira for test management and just report on test results in Confluence. There are, however, multiple Jira add-ons for making product managers’ and testers’ life easier.

In our experience, the four big outstanders within this sector are Xray by Xpand IT, Zephyr, TM4J by Adaptavist and SynapseRT by Go2Group.

Every plugin has its positives and negatives. We and our customers have had the most success with Xray and that’s what we will focus on today.

What is Xray Test Management for Jira?

Xray is “a complete test management tool for Jira”. This means that a project manager can define Robot Framework test cases that the test developers should start writing. Xray uses the underlying Jira Issues system which means all the usual Jira functionalities like custom fields, comments, JQL and Jira REST API work as they would with everything else in Jira.

Xray issue types

When Xray is installed in Jira, five new issue types are created:

• Precondition
• Test Plan
• Test Set
• Test
• Test Execution

The issue types can be used in Jira projects where those have been added to the issue type scheme. To define which projects within Jira are to be used to store requirements, Jira project admin can enable “Requirement Coverage” in the project configuration view. Requirement issue types are defined in Xray Issue Type Mapping configuration.

• Precondition issue type
  Defines what must be done before test steps can be executed
• Test Plan issue type
  Defines what tests should be executed for what version
• Test Set issue type
  A group of tests, for example all tests related to some application functionality. One test can belong to multiple test sets.
• Test issue type
  A test case and it includes test steps, actions and expected results from those
• Test execution issue type
  Represents a single execution of the test. Test executions can be created directly via Jenkins integration

How to use Xray in Jira

Step 1: Installing the Xray plugin

Install the plugin by following these instructions: Xray installation. Without a license, the Xray panels won’t appear in Jira UI so remember to follow the license instructions as well!

The basic usage goes as follows:

• Test designer writes down a test suite using Issue Type User Story
• User Story: Should log in successfully
• Under this User Story, a single test case is written as an issue. This issue has the Issue Type of Test. You should create these test issues inside a user story view so that you can link them together more easily
• As the developer gets tasked with writing the robot test, they will add as a test case tag the unique key of the Jira User Story that defined the test suite that needs to be written
• The developer pushes the changes along with the test cases to version control and the CI server fetches the code
• The CI server (Jenkins in our case) saves the robot results XML file and sends the file to Jira through the Xray integration
• The integration creates a test issue in Jira below the User Story defined in the tag for every test case
• Every subsequent test run creates new issue items that describe the result (pass/fail) of the single test case

The existing tests are structured within a Jira project where Xray is enabled:

Xray also offers reporting tools for tracking requirement coverage, test executions and historical data. Test plan, test execution, test run and traceability reports can also be exported to CSV.

For a deep dive on how to use Xray in Jira, check out this seminar by Xpand IT.

Here are the topics covered:

1. Configuring Xray

2. Working with Requirements

3. Creating different types of tests

4. Creating a pre-condition and using it in a test

5. Different approaches for managing test library in Xray

6. Creating a test plan and test executions

7. Executing manual and automated tests

8. Reporting capabilities

9. Integration with Xporter add-on

How do I integrate Xray with Jenkins and Robot Framework?

Installing Xray connector to Jenkins

Integration with Jenkins Documentation Page

1. Download the plugin .hpi file. On the Integration with Jenkins Confluence page you can find the links to the latest releases. Go to a release and download the .hpi file.

2. Open the Jenkins GUI. You will need admin privileges.

3. Navigate to Manage Jenkins. Open the Plugin Manager. Click on the Advanced tab on the plugin navigator. Scroll down to find the Upload Plugin.

4. Upload the xray-for-jira-connector.hpi file you downloaded in step 1. You may need to restart the server after installation.

5. After restarting the server, navigate to the plugins page to verify that Xray is installed.

6. After the plugin installation is done, it’s time to connect Jenkins and the Jira server together. Navigate to Jenkins > Manage Jenkins > Configure System.

7. Find the Xray configuration section. In the Configuration alias you need to enter a name for the configuration. The server address is your Jira instance’s address. The username and password must exist in the Jira instance and they must have Create Test and Test Execution Issues privileges.

Using Jenkins, Xray and Robot Framework together

1. Use the video seminar to familiarize yourself with the issue creation (section 2: create different types of tests)

2. In Jira: Create a User Story which defines the test suite you want to be done. After that issue has been created, open that user story and click the “create test” button. After you have done that, write down the unique ID of the User Story.

3. In Robot: Add User Story ID’s as tags to your robot test. One tag should be reserved for one User Story/test suite ticket: Xray - Robot XML Report Documentation Page.

4. In Jenkins: Use Jenkins Pipeline Syntax Generator to create a Step for Jenkinsfile’s post run operations. Using this step you should add the unique project ID to your Jenkinsfile. Below is an example post step including artifact saving, robot reporting and lastly Xray Importing (sending the Robot XML to Jira through Xray Jenkins plugin). In this snippet, the project key is EDS. The server Instance comes from the Jenkins configuration page’s settings (“configuration ID”). Here’s a pipeline step example.

5. After you have run the build and robot tests, you should go back to Jira and check out the User Story you created. Under that user story, you should find the “test” issue you created, and next to that issue you should see the automatically created issues. Each issue is for one keyword and you can find all the keys in a single robot file under a “test execution”

Would you like to learn more about test automation in software development?

My colleagues at Eficode have written multiple great blog posts on testing in Software world and why it is crucial, here’s for example a blog on (Acceptance) Test-Driven Development, and about test automation maturity self-assessment.

The biggest benefit of having your test cases in Jira is the fact that you can have requirements, bugs, and stories all in one system. With the help of Jira’s powerful search functionality, you will most likely find why and how something was done. By using Jira to also manage your helpdesk, the bug reporting, fixing, and testing will also remain within a single application.

Xpand IT has been working on Xray meticulously and pushing out new features consistently.

We’ll be following Xray closely and keeping an eye out for even more integrations for automated testing and other cost-saving improvements!