How to create and manage test cases with Xray and Jira

What is a test?

A test is an abstraction of a test idea/scenario and a way to verify/validate associated requirement(s). It’s one or more checks, an experiment, or an investigation with the goal of obtaining information about a quality attribute of the test target.

A traditional, manual test involves a sequence of steps coupled with conditions or variables, test inputs, and expected results. It is intended to establish the quality, performance, and/or reliability of a piece within a system. 

Usually, every requirement or objective of the test target is expected to achieve at least one test.

The success of a test is determined by comparing the expected and actual results.

 A test can be: 

• Scripted (e.g. test case or automated test) or exploratory 
• Specified using Gherkin (scenario)
• Executed manually or through automation
• Linked to/cover one or more requirements
• Like any other Jira issue type, it can be labeled, prioritized, assigned to components, and commented on

Tests and testing in general don’t ensure quality. They provide information about quality.

100 percent coverage is a myth. Adding more tests doesn’t ensure better information about the quality that matters to you.

It’s often better to focus on assessing what stakeholders are concerned about, consider risks, and specify and perform testing based on these metrics. 

There will always be some bugs. It’s important to understand what’s valuable so testing can focus on what threatens those values.

Test process in Xray

Xray allows you to plan, design, and execute tests, as well as generate test reports. Xray uses specific Jira issues types for this process. 

Each testing phase allows you to use the following issues:

• Plan phase: Test plan issues
• Design phase: The specification is defined using precondition and test issue types. The organization of the tests can be defined using test sets
• Execute phase: Test execution issues
• Report phase: Test execution issues including built-in requirement coverage reports and custom issues using Jira Software tools

To start, it’s useful to add tests to a small project. You can use test issues to create tests for your requirements and execute ad hoc, unplanned test executions. Here’s how:

Step 1: How to write and specify tests

Test types

There are three default test types:

• Cucumber: Write tests in a business-readable domain and specific language (Gherkin)
• Generic: An unstructured test, without steps. This is a way to abstract and have visibility of traditional automated tests or exploratory tests
• Manual: A traditional test case composed of a list of steps that are scripted

Each test type has one of three kinds of tests:

• Gherkin
• Unstructured
• Steps

Actions you can perform on Xray tests: 

• Create
• Edit
• Clone

Create

To create a new test issue follow these steps: 

Step 1: Click Create Issue at the top of the screen to open the Create Issue dialog box/page.

Step 2: How to organize tests

Organizing tests is a crucial part of daily testing activities. Since you might start with a large number of tests and your tests may grow over time, it’s important to organize them in an efficient and effective way that works for you. 

Xray provides two different ways to organize your tests:

1. Use the test sets issues
2. Use the test repository

The test set is a simple way to create different groups of tests, since it is a flat list of tests. You may have as many test sets as you wish and a test may be included in multiple test sets. Test sets are ideal if you want to have full control over certain groups of tests.

Create test sets

To create a new test set issue:

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog/page.

Step 2: Select the Project and on Issue Type, select Test Set.

Step 3: Type a Summary for the test set and complete at least all fields marked by an asterisk.

Step 4: When you are satisfied with the content of your test set, click the Create button.

The test repository is a tree-like organizational structure at the project level. It allows you to hierarchically organize tests within folders and sub-folders. This folder concept is common in some tools and resembles file organization in a computer's operating system.

Create a test repository

To create a test repository, create a new folder:

Step 1: Click on the parent folder.

Step 2: Click on the add icon at the top of the folders section.

Step 3: How to plan tests

Test planning allows you to decide your testing strategy, including the issues you want to validate, how to validate them, if tests will be manual or automated, how resources will be allocated, and when and who will execute the tests.

Depending on several factors, you may want to prioritize some tests or focus on specific requirements.

Create a test plan

Here’s how to create a new test plan Issue: 

Step 1: Click Create on the top of the screen to open the Create Issue dialog box/page or press the "C" key on the keyboard.

Step 4: How to execute tests

Test execution occurs between developing test scripts and reporting and analyzing test results. 

A test execution is an issue type that aggregates a user-determined collection of tests. It monitors and verifies if tests work as expected in a target context and environment. The overall execution status, updated after each test is performed, informs you about the progress of the test execution, including which tests passed, failed, are being executed, or are waiting to be performed.

Create a test execution issue

Step 1: Click the + at the left of the screen to open the Create Issue dialog box.

Learn more

For more detailed information on how to create and manage test cases in Jira and Xray, check out the Xray Academy.

See more integrations for Atlassian Open DevOps.