Introduction

I'm Kobayashi, a Product Manager (PdM) for internal systems at KINTO Technologies. After joining the company, I was assigned to the website restructuring project of KINTO ONE ([KINTO] New Vehicle Subscription from Toyota | Full Service Leasing (kinto-jp.com)), and was in charge of project manager, test promotion, and migration promotion. When I was assigned to the project, it was already a year into development, and we were at the stage of conducting integration tests and releases. However, we encountered a number of challenges afterwards. In this article, I would like to introduce some of the challenges I first encountered as a person in charge of test promotion.

What is the Website Restructuring Project?

It’s the name of an in-house development project to renew our KINTO ONE New Vehicle subscription e-commerce site, our main KINTO service in Japan. The goal was to improve development productivity through a 360 review of its architecture and data structures. There were more than 20 people involved from different products and services, and a total of about 50 window persons from the business and development sides.

First, Grasp the Situation

I joined KINTO Technologies in February 2022. The first step was to participate in the regular weekly meetings, gaining an understanding of the situation and taking a stance on implementing testing and transition promotion.

The internal integration test was started in mid-January 2022, shortly before I joined the company. It was reported that the project was on track, with a slight delay of a few days. At this point, there was no incongruity in the schedule and the internal integration test was scheduled for completion at the end of May 2022. It was an ideal situation where developers created the internal integration test plan, and a test team of non-developers was responsible for writing and executing test cases.

Initial Challenges

The internal integration test was planned to be divided into 7 phases, but the progress began to slow down in late February, and the first report in March was delayed by a week. At that time, there were more than 10 bugs that affected the subsequent testing phase. It became apparent that it was a difficult situation where it would take about a week just to fix them. To confirm if we could proceed as planned, we interviewed the developers about the situation, and the following comments came up.

• Not aware of the completion conditions for unit test
• Not aware of start conditions for integration test
• The content of the internal integration test differed from expectations

Although the document compiled by the developers describes what kind of tests to be conducted, there is certainly no description of the start and completion conditions. That kind of information is being sought in written policy and plans, but it cannot be found. Well, it made sense.

Regarding the expected test content, the document compiled by the developers stated that the test would be conducted through a series of screen transitions, and the test items were described as follows.

• Is the screen display and transitions correct when the correct data is entered?
• Are records correctly created in the DB?
• Is it possible to recover data when the process is interrupted in the middle?

From this content, it seems that story-based testing is assumed, aligning well with the test cases prepared by the test team. Consequently, we had to consider what caused the confusion. When you take a look at the documentation compiled by the developers, you will notice the following description about how to run the tests.

• Conduct testing on each browser (Same as screen test)
• For the database, directly check the DB values using SQL

Stated that it is the same as the screen test conducted in the unit test. In this description, it appears that same tests as the unit test are conducted by integrating the front end and back end. I somehow understood the cause of the confusion. This challenge emphasizes the importance of planning without inconsistencies in understanding.

Addressing Challenges

The response to this challenge was to stop the internal integration test for 2 weeks, since forcing the test to proceed without stopping could risk further delays. This response had the following effects:

Recovery of the Development Side is Possible

Stopping the test allowed time to be used for correction and delayed recovery. It also increases quality because it allows us to focus on correction. Developers and the test team are freed from the stress of not progressing testing.

Quality Control Policy and Quality Evaluation Criteria Can be Organized

Stopping the test made it possible to organize the policy and criteria, not only for the internal integration test, but also for the entire test, enabling the dissemination of the following information.

• Define the quality to be ensured in each test phase as a quality control policy
• Define quality evaluation criteria in each test phase

Although it was a postscript policy and criteria, the developers accepted it. This flexibility is one of our strengths. The internal integration test began to run successfully.

What To Do After

Various challenges will continue to arise thereafter. When schedule changes occurred due to other projects, we added quality enhancement tests and revised the plan to improve quality according to the situation, while conducting external integration tests, follow-up intake of other projects, and QA testing. As a result, we successfully released it in August 2023. It is said that, given its size, there are few post-release bugs. Nevertheless, when a bug occurs, it can significantly impact business operations. Therefore, I would like to continue exploring what I can do to improve quality.

Conclusion

Based on this experience, I believe the following 2 points are important.

• Plan without inconsistencies in understanding
• Establishment of quality control policy and quality evaluation criteria
  I thought that even if it is left to the developers, it's essential to have plan, policy, and criteria in place as a guidepost in case something happens.

Also, what I realized while responding to the issue this time is that we never gave work instructions to the developers. In the website restructuring project, the policy was to leave everything from detailed design to internal integration test to the developers. This stands as a project policy that I wanted to uphold. I think I managed to do so.

Well, these were some of the initial challenges I encountered in the website restructuring project.