Implementing Screenshot Testing in the Unlimited Android App Was Tougher Than Expected

This article is the entry for day 13 of the KINTO Technologies Advent Calendar 2024 🎅🎄

Hello! My name is fabt and I am developing an app called Unlimited (its Android version) at KINTO Technologies. I have recently implemented a screenshot test, which has been much talked about, so I will introduce the steps of it, the stumbling blocks I encountered and how to solve them. I hope this will be helpful for those who are considering implementing a screenshot test in the near future.

What is Screenshot Testing?

Screenshot testing is a method where the app takes screenshots based on the current source code in development and compares them to previous versions to confirm and verify changes. Roughly speaking, it can even detect a 1dp difference that is hard for the human eye to notice, making it easier to ensure no unintended UI modifications slip through. Many conference sessions, including one at DroidKaigi 2024, have addressed this topic, and I sense it's gaining significant attention lately.

Screenshot Test Library Selection

I decided to use Roborazzi, a tool featured in numerous conference presentations and already implemented in several Android apps within our company.

Trying it out

Following the official setup guide and information on the Internet, I'll try to run it in a local environment. Since I'm not doing anything special at this stage, I'll move forward quickly.

First, install the necessary libraries for running the screenshot test.

• Version Catalog (libs.versions.toml)

[versions]
robolectric = "4.13"
roborazzi = "1.29.0"

[libraries]
androidx-compose-ui-test-junit4 = { module = "androidx.compose.ui:ui-test-junit4" }
androidx-compose-ui-test-manifest = { module = "androidx.compose.ui:ui-test-manifest" }
robolectric = { module = "org.robolectric:robolectric", version.ref = "robolectric" }
roborazzi = { module = "io.github.takahirom.roborazzi:roborazzi", version.ref = "roborazzi" }
roborazzi-compose = { module = "io.github.takahirom.roborazzi:roborazzi-compose", version.ref = "roborazzi" }
roborazzi-junit-rule = { module = "io.github.takahirom.roborazzi:roborazzi-junit-rule", version.ref = "roborazzi" }

[plugins]
roborazzi = { id = "io.github.takahirom.roborazzi", version.ref = "roborazzi" }

• root build.gradle.kts file

plugins {
    alias(libs.plugins.roborazzi) version libs.versions.roborazzi apply false
}

• module build.gradle.kts file

plugins {
    alias(libs.plugins.roborazzi)
}

android {
    testOptions {
        unitTests {
            isIncludeAndroidResources = true
            all {
                it.systemProperties["robolectric.pixelCopyRenderMode"] = "hardware"
            }
        }
    }
}

dependencies {
    // robolectric
    testImplementation(libs.androidx.compose.ui.test.junit4)
    debugImplementation(libs.androidx.compose.ui.test.manifest)
    testImplementation(libs.robolectric)

    // roborazzi
    testImplementation(libs.roborazzi)
    testImplementation(libs.roborazzi.compose)
    testImplementation(libs.roborazzi.junit.rule)
}

You should now have successfully added the necessary libraries. Now, let's create a test class like the one below and run it locally!

import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.ui.test.junit4.createComposeRule
import androidx.compose.ui.test.onRoot
import androidx.test.ext.junit.runners.AndroidJUnit4
import com.github.takahirom.roborazzi.DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH
import com.github.takahirom.roborazzi.captureRoboImage
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.annotation.GraphicsMode

@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ScreenShotTestSample {
    @get:Rule
    val composeTestRule = createComposeRule()

    @Test
    fun sample() {
        composeTestRule.apply {
            setContent {
                MaterialTheme {
                    Surface {
                        Text(text = "screen shot test sample")
                    }
                }
            }
            onRoot().captureRoboImage(
                filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/sample.png"
            )
        }
    }
}

By running the following screenshot capture command in the Android Studio terminal, the Text(text = XXXX) component's screenshot should be output as a PNG file.

./gradlew recordRoborazziDebug

Surprisingly, there was no output. This was an unexpected result.

Test Gets Skipped

Although the command execution was successful, for some reason the screenshot result was not output. While investigating, I ran the unit test locally and noticed that the following was displayed:

For some reason, the test events were not received, causing the test to be skipped. If that is the case, it certainly makes sense that the command was successful but there was no output.

After further investigation, it seemed that the problem was a conflict in JUnit versions. Our app typically uses Kotest for unit testing, which runs on JUnit 5, while Roborazzi (or more specifically, Robolectric) uses JUnit 4 as its test runner. Surprisingly, there were similar issues in Roborazzi's github issues.

The solution was to use the junit-vintage-engine, a library as mentioned in the above issue. Briefly speaking, this library allows JUnit 4 and JUnit 5 to coexist and is sometimes used for migration.

Now, let's add junit-vintage-engine, run the command again to see if it works.

First, add dependencies (the parts introduced in the previous section are omitted)

• Version Catalog (libs.versions.toml)

[versions]
junit-vintage-engine = "5.11.2"

[libraries]
junit-vintage-engine = { module = "org.junit.vintage:junit-vintage-engine", version.ref = "junit-vintage-engine" }

• module build.gradle.kts file

dependencies {
    testImplementation(libs.junit.vintage.engine)
}

Run the screenshot save command again. This time, the Text(text = XXXX) screenshot should finally be output as a PNG file.

Execution result

Initialization Failure

It seems that the execution failed. Before adding the library, it couldn't even be executed, so let's take it as a positive step forward and aim to solve it honestly. It is a process of trial and error.

As the first step in troubleshooting, I checked the logs and found the following output.

It seems like something failed during initialization.

When I check the output logs, it seemed that it was trying to initialize our app's Application class. After a closer look at the Robolectric configuration, I found the following statement:

Robolectric will attempt to create an instance of your Application class as specified in the AndroidManifest.

Robolectric appears to create an instance of the Application class specified in AndroidManifest.xml . This matches what I saw in the logs!

To prevent initialization failures, I'll follow the official documentation and use the @Config annotation to specify a simple Application class.

import android.app.Application
import org.robolectric.annotation.Config

@RunWith(AndroidJUnit4::class)
@Config(application = Application::class) // Add
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ScreenShotTestSample {

With high expectations, run the screenshot save command (omitted) from the Android Studio terminal. Surely, this time the screenshot of Text(text = XXXX) should be output as a png file.

A screenshot successfully generated!

To try out the comparison (screenshot test), I made a slight change to the test class and ran the command.

// Text(text = "screen shot test sample")
Text(text = "compare sample") // Change the text

./gradlew compareRoborazziDebug

The comparison result has been successfully generated!

Comparison image

Comparison result

Summary of Success So Far

I created a simple test class and was able to run a screenshot test without any problems. While adjusting the details of unit tests and property specifications can be challenging depending on the project, the ability to visually confirm changes makes it clear and fast. Therefore, if possible, implementing screenshot testing can help maintain higher app quality by making unintended UI changes easier to detect.

Side Note

At this time, we also implemented CI for our app. Using the GitHub Actions Workflow, we achieved the point of saving, comparing, and commenting screenshots on a pull request. I'll omit the details since we didn't do anything particularly technical, but we have chosen a method that stores the results using the companion branch approach as described in official documentation. (We have integrated yml files, etc., but only at a general level of optimization for smoother implementation.)

Let's get back to the topic.

Including Preview Functions in Testing

When implementing Composable functions, it's common to create corresponding Preview functions as well. If you implement tests manually as in the first half of this article,

1. Composable Implementation
2. Preview implementation
3. Test Implementation

It's quite a hassle... So, let's make the Preview function the target of screenshot testing! That's exactly what this section is about! This is another technique that has been a hot topic.

Anyway, try it out

The procedure is generally as follows:

1. Collect Preview functions
2. Run screenshot tests on the collected functions

It's simple.

There are several ways to collect Preview functions, but this time, I'll use ComposablePreviewScanner.

The main reason I chose this is that it has well-documented setup steps, making investigation easier. Additionally, it seems likely to be officially integrated and supported in the future, though it's still experimental at the time of writing.

Now, let's install the necessary libraries.

• Version Catalog (libs.versions.toml)

[versions]
composable-preview-scanner = "0.3.2"

[libraries]
composable-preview-scanner = { module = "io.github.sergio-sastre.ComposablePreviewScanner:android", version.ref = "composable-preview-scanner" }

• module build.gradle.kts file

dependencies {
    // screenshot testing(Composable Preview)
    testImplementation(libs.composable.preview.scanner)
}

Next, let's create a test class by referring to the README of composable-preview-scanner and insights from those who have used it before.

import android.app.Application
import androidx.compose.ui.test.junit4.createComposeRule
import androidx.compose.ui.test.onRoot
import com.github.takahirom.roborazzi.DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH
import com.github.takahirom.roborazzi.captureRoboImage
import com.kinto.unlimited.ui.compose.preview.annotation.DialogPreview
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.ParameterizedRobolectricTestRunner
import org.robolectric.annotation.Config
import org.robolectric.annotation.GraphicsMode
import sergio.sastre.composable.preview.scanner.android.AndroidComposablePreviewScanner
import sergio.sastre.composable.preview.scanner.android.AndroidPreviewInfo
import sergio.sastre.composable.preview.scanner.android.screenshotid.AndroidPreviewScreenshotIdBuilder
import sergio.sastre.composable.preview.scanner.core.preview.ComposablePreview

@RunWith(ParameterizedRobolectricTestRunner::class)
class ComposePreviewTest(
    private val preview: ComposablePreview<AndroidPreviewInfo>
) {
    @get:Rule
    val composeTestRule = createComposeRule()

    @Config(application = Application::class)
    @GraphicsMode(GraphicsMode.Mode.NATIVE)
    @Test
    fun snapshot() {
        val fileName = AndroidPreviewScreenshotIdBuilder(preview).ignoreClassName().build()
        val filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/$fileName.png" // Preview function name.png

        composeTestRule.apply {
            setContent { preview() }
            onRoot().captureRoboImage(filePath = filePath)
        }
    }

    companion object {
        private val cachedPreviews: List<ComposablePreview<AndroidPreviewInfo>> by lazy {
            AndroidComposablePreviewScanner()
                .scanPackageTrees(
                    include = listOf("XXXX"), // Set up a package to look for Preview functions
                    exclude = listOf()
                )
                .includePrivatePreviews() // Include private Preview functions
                .getPreviews()
        }

        @JvmStatic
        @ParameterizedRobolectricTestRunner.Parameters
        fun values(): List<ComposablePreview<AndroidPreviewInfo>> = cachedPreviews
    }
}

Now, execute the screenshot save command.

...However, it took a lot of time and even after an hour, it showed no signs of finishing. With fewer than 200 Preview functions, I didn't expect it to take this long. As expected, it wasn't going to be that easy.

Test Execution Taking Too Long

After doing some research, I found similar cases.

• https://github.com/takahirom/roborazzi/issues/388

A single test took about 5 minutes to complete, but after removing CircularProgressIndicator, the execution speed significantly improved. Digging deeper into the issue discussion, it seems that it takes a long time to test a Composable that includes infinite animation. As a solution, one suggested setting mainClock.autoAdvance = false to stop automatic synchronization with Compose UI and manually advancing time instead. By manually controlling time, we can capture screenshots at any time, avoiding the impact of infinite animations. Since our app also uses CircularProgressIndicator, this is definitely worth trying! I'll implement it immediately.

I added mainClock. XXXX to the test class.

1. Pause time.
2. Advance by 1,000 milliseconds (1 second).
3. Capture the screenshot.
4. Resume time. That's the flow.

composeTestRule.apply {
    mainClock.autoAdvance = false // 1
    setContent { preview() }
    mainClock.advanceTimeBy(1_000) // 2
    onRoot().captureRoboImage(filePath = filePath) // 3
    mainClock.autoAdvance = true // 4
}

(On a separate note, there's a library called coil for loading images asynchronously, which may not be tested correctly either. ](https://github.com/takahirom/roborazzi/issues/274)But since the solution seems to be the same, we should be able to handle these together.)

Alright, execution time!

The test execution became much faster, but it failed.

Failure When Dialogs Are Included

Although the execution speed has increased, there is no point if it fails. In situations like this, we can always rely on logs.

ComposePreviewTest > [17] > snapshot[17] FAILED
    java.lang.AssertionError: fail to captureRoboImage
    Reason: Expected exactly '1' node but found '2' nodes that satisfy: (isRoot)
    Nodes found:
    1) Node #534 at (l=0.0, t=0.0, r=0.0, b=0.0)px
    2) Node #535 at (l=0.0, t=0.0, r=320.0, b=253.0)px
    Has 1 child
        at androidx.compose.ui.test.SemanticsNodeInteraction.fetchOneOrThrow(SemanticsNodeInteraction.kt:178)
        at androidx.compose.ui.test.SemanticsNodeInteraction.fetchOneOrThrow$default(SemanticsNodeInteraction.kt:150)
        at androidx.compose.ui.test.SemanticsNodeInteraction.fetchSemanticsNode(SemanticsNodeInteraction.kt:84)
        at com.github.takahirom.roborazzi.RoborazziKt.captureRoboImage(Roborazzi.kt:278)
        at com.github.takahirom.roborazzi.RoborazziKt.captureRoboImage(Roborazzi.kt:268)
        at com.github.takahirom.roborazzi.RoborazziKt.captureRoboImage$default(Roborazzi.kt:263)
        at com.kinto.unlimited.ui.compose.ComposePreviewTest.snapshot(ComposePreviewTest.kt:49)

There were multiple logs like the one above. It said there were two nodes. Searching based on the log information, I found an interesting issue. As it's related to compose-multiplatform, the cause may be different, but it seems that the failure is occurring in sections where Dialog()Composable is being used.

When I was wondering if I had to give up, I discovered that an experimental function has been added to capture images that include dialogs! Since it's experimental, using it for all tests might not be ideal, but if I can determine whether the test target is dialogs, it may succeed.

So, I decided to create a custom annotation called DialogPreview(), applied it to the Previews that include dialogs, and modified the test class to retrieve and determine the information.

annotation class DialogPreview()

@OptIn(ExperimentalRoborazziApi::class) // Add
@Config(application = Application::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
@Test
fun snapshot() {
    val isDialog = preview.getAnnotation<DialogPreview>() != null  // Add

    composeTestRule.apply {
        mainClock.autoAdvance = false
        setContent { preview() }
        mainClock.advanceTimeBy(1_000)
        if (isDialog) { // Add
            captureScreenRoboImage(filePath = filePath)
        } else {
            onRoot().captureRoboImage(filePath = filePath)
        }
        mainClock.autoAdvance = true
    }
}

companion object {
    private val cachedPreviews: List<ComposablePreview<AndroidPreviewInfo>> by lazy {
        AndroidComposablePreviewScanner()
            .scanPackageTrees(
                include = listOf("・・・"),
                exclude = listOf()
            )
            .includePrivatePreviews()
            .includeAnnotationInfoForAllOf(DialogPreview::class.java) // Add
            .getPreviews()
    }
}

Determine if the Preview has a DialogPreview annotation (not null) and use captureScreenRoboImage() for dialogs.

Let's run it.

Due to the nature of the Preview function, images and file names are masked.

After loading the Preview functions, I successfully saved their screenshots!

The screenshot comparison was already confirmed earlier, so everything seems to be working fine as it is.

Thank you for sticking with me through this long process!

Conclusion

In this article, I walked through the process of implementing screenshot testing, highlighting the challenges I encountered, and how I solved them. At first, I thought it would be as simple as adding a library and running the tests, but things didn't work out. After a lot of trial and error, I finally got screenshot testing to work. The solutions, such as adding libraries or applying annotations, were relatively simple. However, finding the right information to reach those solutions turned out to be more challenging than I expected. I hope this article will be of some help to you.