How We Implemented Zephyr and TestLink for Seamless JIRA Integration

Hello. My name is Zume, and I am the group manager of the Quality Assurance (QA) Group at KINTO Technologies.

Although I have a long and extensive history in QA, I haven’t been particularly focused on sharing my experience or knowledge until now. However, I thought it would be a good idea to take some time to gather my thoughts, but before I knew it, 2022 came to an end with the ringing of the bells on New Year's Eve.

It's tough to find time for myself when I’m usually busy with work. This has always been my excuse for not making time for personal projects. If I keep saying "I'll do it next month" a few more times, I'll soon find myself welcoming a new year.

About test management

This time, I would like to introduce the benefits of the test management tools used by my group and the journey we took to implement them.

To all the QA engineers reading this article, how are you managing your test cases? Some of you may already be using some kind of paid test management tool.

Generally, Excel or Spreadsheets tend to be used for managing test cases and test executions. However, when using Excel or Spreadsheets for test management, we encountered several challenges:

Challenges in the test process

- Issues
  - ⇒ Concerns (potential issues)

1. Test case structuring often becomes personalized by the test designers, and case classifications and formats vary.

   ⇒When the designer changes, the handover process becomes complicated

   ⇒Due to the lack of standardized format, it takes time to understand cases when the project changes.

2. To review the cases, you need to open and check the contents of files each time.

   ⇒It is difficult to share documents and know-how within the team.

3. Stakeholders (other than QA) have a hard time getting an overview of the test content and results.

   ⇒The QA side needs to prepare reports for stakeholders.

4. For regression testing, a new file needs to be created for each test cycle.

   ⇒It becomes difficult to track which cases were reused.

5. It is difficult to follow the change history or update history of test cases.

   ⇒Maintenance, including case updates, takes a lot of time (plus Excel is not suitable for simultaneous online editing by multiple users)

6. Since the test execution results are entered manually, the exact execution time is unknown.

   ⇒It is challenging to pinpoint the exact time when defects occur

7. Test cases and bug reports are not linked

   ⇒It becomes difficult to compile statistics such as the defect rate for each function (manual compilation is possible but very tedious).

And so on.

To address these challenges, we considered implementing tools that support a series of test activities such as test requirements management, test case creation, test execution, and test reporting. In fact, we never considered using Excel or Spreadsheets from the beginning.

This is because we knew from our experience that once Excel-based operations become ingrained, it takes a lot of time to shift away from them.

Evaluation of tools to be implemented

Initially, the tools we considered were:

• TestLink : An open source web-based test management tool. Free of charge.
• TestRail : A web-based test management tool. Paid.
• Zephyr for JIRA : A JIRA plugin. Paid. (Renamed to Zephyr Squad in 2021^[1])

One of the reasons we considered TestLink was my experience with it at my previous workplace. Another advantage is that it can be tested right away by installing Docker even in a local environment. In fact, I once used a Mac for both testing and running TestLink.

However, I joined KINTO Technologies in March 2020 (when it was still KINTO Co., Ltd.), and the project for which we planned to introduce the tool was scheduled to be released two months later, in May. To make things more challenging, the first state of emergency due to the spread of the new COVID-19 was declared in April during this period.

In such a nerve-wracking situation, which tool did we choose as the most appropriate option?

It was Zephyr for JIRA.

The biggest advantage was that it could be implemented quickly as an add-on for JIRA, which was already being used within the company. Additionally, considering the unexpected shift to remote work during the COVID-19 pandemic, it was convenient since it could be accessed from outside the company.

Although it was a paid tool, we decided to start using it with the idea that if we could get through the May release, we would reassess its continued use.

Looking back at my notes from that time:

Since it's a JIRA plugin, I thought I could change the language settings, but it seems only parts of it support Japanese.

Zephyr's reports are based on scenarios, and there is no reporting function for individual test steps.

etc. ^[2]

These notes reflect our trial and error process. It brings back all the memories.

Although it was easy to implement, it is still essential for users to be familiar with the system and possess the necessary skills. In that sense, I am still grateful to the team members who flexibly navigated that chaotic period with me.

Using Zephyr

It's been almost three years(!)Even though it’s an old story, here are my impressions of using Zephyr for JIRA.

As it is a JIRA plugin, test cases can be created in the same way as normal issues by selecting the desired issue type. Case items include steps, expected values, results, status, comments, and file attachments, making it convenient to leave screen captures as evidence for each step.

On the other hand, it took quite a long time to load the plugin itself. The problem was that it took a few seconds each time we changed screens. A similar question for help was posted on the Atlassian community, so it may be a Zephyr-specific issue .

And now to TestLink

Now, let's talk about test management after we somehow managed to meet the release schedule and handed off the project in May 2020.

We reconsidered the cost aspect as well. Assuming the tool is linked to JIRA and the number of users is around 10 to 20 people,

the prices as of 2020 were as follows:

Zephyr for JIRA: 11-100 Users ¥510/user/month ⇒ ¥10,200/month for 20 users
TestRail: 1-20 Users $32/user/month ⇒ $640 (approx. ¥83,200)/month for 20 users

The prices as of 2023 are as follows:

Zephyr Squad: 11-100 Users ¥570/user/month ⇒ ¥11,400/month for 20 users
TestRail: 1-20 Users $37/user/month ⇒ $740 (approx. ¥96,200)/month for 20 users

The fee structure has changed slightly since then, and the prices have gone up a bit. *All prices are calculated at 130 yen to the dollar

At first glance, Zephyr seems like a good deal, but since it is a plugin for JIRA, you will actually need to have the same number of licenses as you do for JIRA. In that regard, since not everyone in the Development Division will use it and only QA members will, we want to avoid increasing costs as the organization expands.

Still, TestRail is quite expensive. Considering the cost, there is no better option than the free TestLink.

Although the UI of TestLink is not the best (it's open source so I won't complain), as a test management tool, it can at least resolve the issues mentioned above as follows.

Testing process challenges and their solutions

So, we decided to introduce TestLink from June 2020 onwards.

Well, I'm sure my teammates will get annoyed if I say it's easy, but the truth is that while the tool isn’t omnipotent, it's a lot easier than using data files like Excel.

Postscript

Even though it's free, there are still infrastructure costs to run it. We are using an AWS instance for TestLink, which costs several tens of thousands of yen per year. It has been almost three years since we started using it, and so far we have been able to operate it without any major issues.

In this article, I explained how we implemented TestLink as a test management tool in the QA group.

In future posts, I hope to discuss how TestLink is used in actual projects, its integration with JIRA, and more.