Key Aspects of QA Testing for Native Apps

Introduction

I am Okapi, from the Quality Assurance Group.
In the previous article, titled "Increasing Awareness of QA ," I described how the QA Group participates in projects and performs QA tasks.
In this article, I would like to write about the key characteristics of QA testing for native apps, drawing from my recent experience working on several native app projects.

About QA Testing for Native Apps

In basic QA design, a test scenario is created that clarifies the scope of the testing (what is to be tested or excluded), and test cases (preconditions, procedures, and expectations) are then created from that perspective. However, in addition to the above, a native app design requires consideration of the following points:

･ iOS and Android
･ OS versions and smartphone devices to be tested
･ Native app functionality

As a result, even native apps with less complex functions exhibit characteristics that tend to increase the scale of testing.

Item	Characteristics
iOS and Android	･ iOS is an operating system developed by Apple ･ Android is an operating system developed by Google Due to the differences in the development environments, the amount of testing simply doubles.
OS versions tested and smartphone devices to be tested	There are many types of smartphone devices available to the public. Testing all of these and OS version combinations is practically impossible.
Native app functionality	Due to the unique functions and specifications of mobile devices affecting app functionality, it is necessary to verify the operation of installed native apps based on these characteristics. ･ Push Notification: 　The function that enables apps to send notifications to the user ･ Touchscreen: 　The feature that allows users to interact with the app by touching areas on the screen ･ Permission: 　The feature that enables a pop-up dialogue requesting the user’s consent before the app attempts to access certain functionalities of the device.

iOS and Android

As the operating systems of iOS and Android are different, the amount of testing is doubled, so we want to reduce the man-hours. However, the development team is also divided into iOS, Android and BFF.

• Even if there are no bugs on the iOS version, bugs may be found on the Android version.
• Same may be true on reverse: no bugs on Android, but issues on the iOS version.
• Issues with the API may require testing on both Android and iOS.

These patterns may occur with each function. Therefore, if we proceed in such a way as to omit testing on Android because it has been confirmed on iOS, or vice versa, there is a risk that 'bugs affecting quality' may persist. So we are proceeding with a basic policy of implementing the system for both operating systems.

OS Versions and Smartphone Devices to be Tested

Since testing all combinations of OS and devices is impossible, we always confirm the recommended environment for the target app (e.g., iOS 15.0 or higher / Android 9 or higher), taking into account the market share of each OS. And then assign:

• Main OS: All tests conducted
• Non-main OS: Display check and locations where OS-related bugs occurred
  As device-dependent bugs are rare, and even if they occur, they are unlikely to be fatal, testing is basically conducted on devices that comply with the above-mentioned OS.
  However, there is an exception: Functional tests involving camera activation will be conducted on all types of devices available to us. The reason for this is that during the initial QA testing of a native app, there was a relatively high frequency of issues where the app would crash when the camera was launched on certain devices.

Native App Functionality

In the process of QA of native apps, it is necessary to verify whether the app operates in accordance with specifications, considering the functionality of the mobile device. However, the specification sheet typically outlines only the app specifications and often does not describe the functions of the mobile device. Therefore, bugs can occur when device functions affect the behavior of the app. Test designs based on an understanding of device functions can effectively detect bugs from a usability perspective.

Item	App Specifications	Device Functions	Common Bugs from a Usability Perspective
Push Notifications	･ Push notifications' 　- timing 　- content 　- transition destination when tapped ･ In the app notification badge's 　- display timing 　- deletion timing	･ Notification badge of app icon 　- display when multiple accumulations 　- display timing 　- deletion timing ･ Specification by change of condition 　- login/logout 　- App foreground/background/lock screen	･ The number of app icons does not match the number of notifications when receiving multiple notifications ･ The app notification badge 　- is not displayed 　- display timing is slow ･ The app notification badge 　- does not disappear 　- deletion timing is slow ･ When logging out, the target notification 　- cannot be received 　- error occurs when notification is pressed ･ No notifications when the app is in the foreground
Touchscreen	Behavior when a button or link is tapped	･ Double-tap 　- tap twice quickly ･ Flick 　- swipe left and right with a fingertip ･ Swipe 　- slowly slide left and right ･ Pinch in 　- touch the screen with two fingers simultaneously and move them closer together to zoom out ･ Pinch out 　- touch the screen with two fingers simultaneously and move them apart to zoom in	･ Double-tap 　- to open the same screen twice 　- registration details are registered twice ･ Display error occurs when performing flick/swipe/pinch in/pinch out
Permissions	ー	･ Allow/disallow push notifications ･ Allow/disallow location information ･ Allow/disallow camera	･ Push notification permission setting is not linked to settings ･ Crash occurs with location permissions and camera permissions

Next Challenges

In QA testing scenarios for native apps, we check for specifications according to system requirements and app-specific functions. As I became more fully involved in QA for app development, I learned that the iOS screening process is very strict.

For example,
･ Rejected unless it is in "non-login accessible" format
･ Rejected if it contains a link to GooglePlay
and so on.

I believe that by adding these insights into our knowledge, we can incorporate them into further ad-hoc testing scenarios as one perspective and lead to further quality improvements.

Conclusion

In this article, I introduced the characteristics of QA testing for native apps that have a tendency to increase the scale of testing. However, in native apps, QA reports numerous bugs from various perspectives, including iOS, Android, and API. This places a substantial burden on the development side to address modifications, resulting in higher costs for both sides compared to web apps, making the current situation less efficient.

The latest app development languages I worked with include Kotlin for Android and Swift for iOS, respectively. Native development for each platform has the advantage of individual optimization that allows for fine-tuning.

Although there are cross-platform technologies such as Kotlin Multiplatform Mobile (KMM) and Flutter that can be developed using the same source code, there are disadvantages, such as the need to learn separately the parts that depend on individual OS, such as device control.^[1]

In that regard, native languages are directly supported by the corresponding OS, providing a high degree of freedom in leveraging device functions.
In the future, I expect and hope that the most suitable development method will be chosen based on the preferences of development engineers, as well as the purpose of app development. As a QA engineer, I also would like to exert effort in flexibly improving the efficiency of QA testing in alignment with the development method.

脚注

1. Camera, GPS, sensor system, etc. ↩︎