July 19, 2024

iOS

Snapshot Testing in the my route App

Hi! I’m Ryomm, developing the iOS app my route at KINTO Technologies. My fellow developers, Hosaka-san and Chang-san, along with another business partner and I, successfully implemented and integrated our Snapshot Testing.

Introduction

Currently, the my route app team is moving towards transitioning to SwiftUI, so we have decided to implement Snapshot Testing as a foundational step. We began this transition by initially replacing only the content, while keeping UIViewController as the base. This approach ensures that the implemented Snapshot Testing will be directly applicable.

Let me introduce the techniques and trial-and-error methods we used to apply Snapshot Testing to an app built with UIKit.

What is Snapshot Testing?

It is a type of testing that verifies whether there are any differences between screenshots taken before and after code modifications. We use the Point-Free library for modifications https://github.com/pointfreeco/swift-snapshot-testing.

While developing my route, we extend XCTestCase to create a method that wraps assertSnapshots as follows: We determined the threshold to be at 98.5% after various trials to ensure that very fine tolerance variances were accommodated successfully.

extension XCTestCase {
  var precision: Float { 0.985 }
  func testSnapshot(vc: UIViewController, record: Bool = false,
                    file: StaticString, function: String, line: UInt) {
    assert(UIDevice.current.name == "iPhone 15", "Please run the test by iPhone 15")
    // SnapshotConfig is an enum that specifies the list of devices to be tested
    SnapshotConfig.allCases.forEach {
      assertSnapshots(matching: vc, as: [.image(on: $0.viewImageConfig, precision: precision)],
                      record: record, file: file, testName: function + $0.rawValue, line: line)
    }
  }
}

The Snapshot Testing for each screen is written as follows.

final class SampleVCTests: XCTestCase {
  // snapshot test whether it is in recording mode or not
  var record = false
  func testViewController() throws {
    let SampleVC = SampleVC(coder: coder)
    let navi = UINavigationController(rootViewController: SampleVC)
    navi.modalPresentationStyle = .fullScreen
    // This is where the lifecycle methods are called
    UIApplication.shared.rootViewController = navi
    // The lifecycle methods starting from viewDidLoad are invoked for each test device
    testSnapshot(vc: navi, record: record, file: #file, function: #function, line: #line)
  }
}

Tips

We need to wait for the data fetched by the API to be reflected in the View after the viewWillAppear method and subsequent methods.

To ensure the Snapshot Testing run after the API data is reflected in View, we have encountered issues where the tests execute too early, causing problems like the indicator still being visible.

Since it is difficult to determine if the data from the API call has been reflected in the view, we will implement a delegate to handle this verification.

protocol BaseViewControllerDelegate: AnyObject {
  func viewDidDraw()
}

In the ViewController class, create a delegate property that conforms to the previously prepared delegate. If no delegate is specified during initialization, this property defaults to nil.

class SampleVC: BaseViewController {
  // ...
  weak var baseDelegate: BaseViewControllerDelegate?
  // ....
  init(baseDelegate: BaseViewControllerDelegate? = nil) {
    self.baseDelegate = baseDelegate
    super.init(nibName: nil, bundle: nil)
  }
  // ...
}

When calling the API and updating the view, for example, after receiving the results with Combine and reflecting them on the screen, call baseDelegate.viewDidDraw(). This notifies the Snapshot Testing that the view has been successfully updated with the data.

someAPIResult.receive(on: DispatchQueue.main)
  .sink(receiveValue: { [weak self] result in
    guard let self else { return }
    switch result {
    case .success(let item):
      self.hideIndicator()
      self.updateView(with: item)
      // Timing of data reflection completion
      self.baseDelegate?.viewDidDraw()
    case .failure(let error):
      self.hideIndicator()
      self.showError(error: error)
    }
  })
  .store(in: &cancellables)

As we want to wait for baseDelegate.viewDidDraw() to be executed, we add XCTestExpectation to the Snapshot Testing.

final class SampleVCTests: XCTestCase {
  var record = false
  var expectation: XCTestExpectation!
  func testViewController() throws {
    let SampleVC =  SampleVC(coder: coder, baseDelegate: self)
    let navi = UINavigationController(rootViewController: SampleVC)
    navi.modalPresentationStyle = .fullScreen
    UIApplication.shared.rootViewController = navi
    expectation = expectation(description: "callSomeAPI finished")
    wait(for: [expectation], timeout: 5.0)
    viewController.baseViewControllerDelegate = nil
    testSnapshot(vc: navi, record: record, file: #file, function: #function, line: #line)
  }
  func viewDidDraw() {
    expectation.fulfill()
  }
}

When there are multiple sets of data to be retrieved from the API that need to be reflected (when calling baseDelegate.viewDidDraw() in multiple places), you can specify expectedFulfillmentCount or assertForOverFulfill.

final class SampleVCTests: XCTestCase {
  var record = false
  var expectation: XCTestExpectation!
  func testViewController() throws {
    let SampleVC = SampleVC(coder: coder, baseDelegate: self)
    let navi = UINavigationController(rootViewController: SampleVC)
    navi.modalPresentationStyle = .fullScreen
    UIApplication.shared.rootViewController = navi
    expectation = expectation(description: "callSomeAPI finished")
    // When viewDidDraw() is called twice
    expectation.expectedFulfillmentCount = 2
    // When viewDidDraw() is called more times than specified, any additional calls should be ignored
    expectation.assertForOverFulfill = false
    wait(for: [expectation], timeout: 5.0)
    viewController.baseViewControllerDelegate = nil
    testSnapshot(vc: navi, record: record, file: #file, function: #function, line: #line)
  }
  func viewDidDraw() {
      expectation.fulfill()
  }
}

If the baseViewControllerDelegate from the previous screen remains active, running the Snapshot Testing across all screens will call viewDidLoad and subsequent lifecycle methods for each test device every time testSnapshot() is invoked. This causes the API to be called multiple times and viewDidDraw() to be executed repeatedly, resulting in multiple calls error.

Therefore, we clear the baseViewControllerDelegate after calling wait().

Frame misalignment on devices

While Snapshot Testing can generate snapshots for multiple devices, we encountered issues where the layout and size of elements were misaligned on some devices.

Misaligned

This issue is caused by the lifecycle of the Snapshot Testing execution. In a Snapshot Testing, it starts loading on one device, and then other devices are rendered by changing the size without reloading. This means that viewDidLoad() is executed only once at the beginning, and for the other devices, it starts from viewWillAppear().

As a solution, create a MockViewController that wraps the viewcontroller you want to test. Override viewWillAppear() to call the methods that are originally called in viewDidLoad().

import XCTest
@testable import App

final class SampleVCTests: XCTestCase {
  // snapshot test whether it is in recording mode or not
  var record = false
  func testViewController() throws {
    // Write it the same way as when calling the screen
    let storyboard = UIStoryboard(name: "Sample", bundle: nil)
    let SampleVC = storyboard.instantiateViewController(identifier: "Sample") { coder in
      // VC wrapped for Snapshot Test
      MockSampleVC(coder: coder, completeHander: nil)
    }
    let navi = UINavigationController(rootViewController: SampleVC)
    navi.modalPresentationStyle = .fullScreen
    UIApplication.shared.rootViewController = navi
    testSnapshot(vc: navi, record: record, file: #file, function: #function, line: #line)
  }
}
class MockSampleVC: SampleVC {
  required init?(coder: NSCoder) {
    fatalError("init(coder: \\(coder) has not been implemented")
  }
  override init?(coder: NSCoder,
                completeHander: ((_ readString: String?) -> Void)? = nil) {
    super.init(coder: coder, completeHander: completeHander)
  }
  override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    // The following methods are originally called in viewDidLoad()
    super.setNavigationBar()
    super.setCameraPreviewMask()
    super.cameraPreview()
    super.stopCamera()
  }
}

Still not fixed・・・

If the rendering is still misaligned, calling the layoutIfNeeded() method to update the frames often resolves the issue.

import XCTest
@testable import App
final class SampleVCTests: XCTestCase {
  var record = false
  func testViewController() throws {
    let storyboard = UIStoryboard(name: "Sample", bundle: nil)
    let SampleVC = storyboard.instantiateViewController(identifier: "Sample") { coder in
      MockSampleVC(coder: coder, completeHander: nil)
    }
    let navi = UINavigationController(rootViewController: SampleVC)
    navi.modalPresentationStyle = .fullScreen
    UIApplication.shared.rootViewController = navi
    testSnapshot(vc: navi, record: record, file: #file, function: #function, line: #line)
  }
}
fileprivate class MockSampleVC: SampleVC {
  required init?(coder: NSCoder) {
    fatalError("init(coder: \\(coder) has not been implemented")
  }
  override init?(coder: NSCoder, completeHander: ((_ readString: String?) -> Void)? = nil) {
    super.init(coder: coder, completeHander: completeHander)
  }
  override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    // Update the frame before calling rendering methods
    self.videoView.layoutIfNeeded()
    self.targetView.layoutIfNeeded()
    super.setNavigationBar()
    super.setCameraPreviewMask()
    super.cameraPreview()
    super.stopCamera()
  }
}

Looks good

Snapshot for WebView screens

There are situations where you may want to apply Snapshot Testing to toolbars to other elements, but not the content displayed in a Webview. In such cases, it is good to separate the part that loads the WebView content from the WebView’s configuration and mock the loading part during tests.

For the implementation, we separate the method that calls self.WebView.load(urlRequest) etc. to display the Webview content from the method that configures the WebView itself.

// Implementation in the VC
class SampleWebviewVC: BaseViewController {
  // ...
  override func viewDidLoad() {
    super.viewDidLoad()
    self.setNavigationBar()
    **self.setWebView()**
    self.setToolBar()
  }
  override func viewDidAppear(_ animated: Bool) {
    super.viewDidAppear(animated)
    **self.setWebViewContent()**
  }
  // ...

  /**
    * Separate the method for configuring the WebView from the method for setting its content
    */
  /// Configure the WebView
  func setWebView() {
    self.webView.uiDelegate = self
    self.webView.navigationDelegate = self
    // Monitor the loading state of the web page
    webViewObservers.append(self.webView.observe(\\.estimatedProgress, options: .new) { [weak self] _, change in
      guard let self = self else { return }
      if let newValue = change.newValue {
        self.loadingProgress.setProgress(Float(newValue), animated: true)
      }
    })
  }
  /// Set content for the WebView
  private func setWebViewContent() {
    let request = URLRequest(url: self.url,
                             cachePolicy: .reloadIgnoringLocalCacheData,
                             timeoutInterval: 60)
    self.webView.load(request)
  }
  // ...
}

Then, in the mock that wraps the VC under test, we make it so that the method that loads the WebView content is not called.

import XCTest
@testable import App

final class SampleWebviewVCTests: XCTestCase {
  private let record = false
  func testViewController() throws {
    let storyboard = UIStoryboard(name: "SampleWebview", bundle: .main)
    let SampleWebviewVC = storyboard.instantiateViewController(identifier: "SampleWebview") { coder in
        MockSampleWebviewVC(coder: coder, url: URL(string: "<https://top.myroute.fun/>")!, linkType: .Foobar)
    }
    let navi = UINavigationController(rootViewController: SampleWebviewVC)
    navi.modalPresentationStyle = .fullScreen
    UIApplication.shared.rootViewController = navi
    testSnapshot(vc: navi, record: record, file: #file, function: #function, line: #line)
  }
}
fileprivate class MockSampleWebviewVC: SampleWebviewVC {
  override init?(coder: NSCoder, url: URL, linkType: LinkNamesItem?) {
    super.init(coder: coder, url: url, linkType: linkType)
  }
  required init?(coder: NSCoder) {
    fatalError("init(coder:) has not been implemented")
  }
  override func viewWillAppear(_ animated: Bool) {
    // Change the method that was called in viewDidLoad to be called in viewWillAppear
    self.setNavigationBar()
    self.setWebView()
    self.setToolBar()
    super.viewWillAppear(animated)
  }
  override func viewDidAppear(_ animated: Bool) {
    // Do nothing
    // Override to avoid calling the method that sets the WebView content
  }
}

Snapshot of the screen that is calling the camera

Call the camera and also take the snapshot of the screen which displays a customized view. However, since the camera does not work on the simulator, it is necessary to find a way to disable the camera part while still being able to test the overlay.

There was also a suggestion to insert a dummy image to make the camera work on the simulator, but it seems too costly to implement this just for the Snapshot Testing of a non-primary screen.

In myroute’s Snapshot Testing, we used mocks to override the parts that handle the camera input and the parts that set up the capture to be displayed in AVCaptureVideoPreviewLayer, so they are not called. This way, the AVCaptureVideoPreviewLayer displays as a blank screen without any input, allowing the customized View to be shown on top.

In the actual implementation, it is written as follows:

class UseCameraVC: BaseViewController {
  // ...
  override func viewDidLoad() {
    super.viewDidLoad()
    self.videoView.layoutIfNeeded()
    setNavigationBar()
    setCameraPreviewMask()
    do {
      guard let videoDevice = AVCaptureDevice.default(for: AVMediaType.video) else { return }
      let videoInput = try AVCaptureDeviceInput(device: videoDevice) as AVCaptureDeviceInput
      if captureSession.canAddInput(videoInput) {
        captureSession.addInput(videoInput)
        let videoOutput = AVCaptureVideoDataOutput()
        if captureSession.canAddOutput(videoOutput) {
          captureSession.addOutput(videoOutput)
          videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue.main)
        }
      }
    } catch { return }
    cameraPreview()
  }

  override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    // Since the camera cannot be used in the simulator, disable it
    #if targetEnvironment(simulator)
    stopCamera()
    dismiss(animated: true)
    #else
    captureSession.startRunning()
    #endif
  }
}

Override them with mocks as follows: Due to the reasons described regarding the frame misalignment issue, we call the methods from viewWillAppear() that were originally called in viewDidLoad().

class MockUseCameraVC: UseCameraVC {
  // ...
  override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    self.videoView.layoutIfNeeded()
    super.setNavigationBar()
    super.setCameraPreviewMask()
    super.cameraPreview()
    super.stopCamera()
  }
}

The cameraPreview() method uses AVCaptureVideoPreviewLayer to display the camera image from the captureSession, but since we override it to have no input, it renders as a white view.

CI Strategy

At the initial stage of introducing Snapshot Testing, we uploaded reference images to a single S3 bucket. During reviews, we downloaded the reference images each time and ran the tests. However, when a view was modified and the reference images were updated simultaneously, there was an issue where tests for other PRs would fail until the PR with the updated reference images was merged.

Existing issues

To address the issue, we created two directories within the bucket hosting the reference images. One directory hosts the images during PR reviews, and once a PR is merged, the images are copied to the other directory. By doing so, we ensure that updates to the reference images do not interfere with the tests of other PRs.

New testing strategy

Useful shells

my route provides four shells for snapshots.

The first one downloads all the reference images for the current screen. This allows the tests to pass locally.

setup_snapshot.sh

Used when switching from the # develop branch
# Example: Sh setup_snapshot.sh
# Clean up the old files from the reference images directory
rm -r AppTests/Snapshot/__Snapshots__/
# Download reference images from S3
aws s3 cp $awspath/AppTests/Snapshot/__Snapshots__ --recursive --profile user

The second shell uploads modified reference images to the PR review S3 bucket when creating a Pull Request.

upload_snapshot.sh

# When creating a PR, upload the modified tests as arguments.
# Example: Sh upload_snapshot.sh ×××Tests
path="./SpotTests/Snapshot/__Snapshots__"
awspath="s3://strl-mrt-web-s3b-mat-001-jjkn32-e/mobile-app-test/ios/feature/__Snapshots__"
if [ $# = 0 ]; then
  echo "No arguments provided"
else
  for testName in "${@}";
  do
  if [[ $testName == *"Tests"* ]]; then
    echo "$path/$testName"
    aws s3 cp "$path/$testName" "$awspath/$testName" --exclude ".DS_Store" --recursive --profile user
  else
    echo "($0testName) No tests found"
  fi
  done
fi

The third shell individually downloads the reference images for the modified screens. It is used when reviewing a Pull Requests that includes screen changes.

download_snapshot.sh

# When reviewing tests, download the reference images for the specific tests
# Example: Sh download_snapshot.sh ×××Tests
if [ $# = 0 ]; then
  echo "No arguments provided"
else
  rm -r AppTests/Snapshot/__Snapshots__/
  for testName in "${@}";
  do
  if [[ $testName == *"Tests"* ]]; then
      echo "$localpath/$testName"
      aws s3 cp "$awspath/$testName" "$localpath/$testName" --recursive --profile user
  else
      echo "($0testName) No tests found"
  fi
  done
fi

The fourth shell forcibly updates the reference images. Although it is basically unnecessary because the reference images for screens with modified test files are automatically copied, it is useful when changes to reference images occur without modifying the test files, such as when common components are updated.

force_upload_snapshot.sh

# If changes affect reference images other than the modified test files, (for example, when common components are updated),
# Please upload manually
# Please use it after merging
# Example: Sh force_upload_snapshot.sh × × × Tests
if [ $# = 0 ]; then
  echo "No arguments provided"
else
  echo "Do you want to forcibly upload to the AWS S3 develop folder? 【yes/no】"
  read question
  if [ $question = "yes" ]; then
    for testName in "${@}";
    do
    if [[ $testName == *"Tests"* ]]; then
        echo "$localpath/$testName"
        aws s3 cp "$localpath/$testName" "$awsFeaturePath/$testName" --exclude ".DS_Store" --recursive --profile user
        aws s3 cp "$localpath/$testName" "$awsDevelopPath/$testName" --exclude ".DS_Store" --recursive --profile user
    else
        echo "($testName) No tests found"
    fi
    done
  else
    echo "Termination"
  fi
fi

Since having four shells can be confusing regarding when and who should use them, we have defined them in the Taskfile and made the explanations easily accessible. When executing, we have to use -- when passing arguments such as specifying file names, making the command bit longer. As a result, we often call the shells directly. However, having this setup is valuable just for the sake of clear explanations.

Taskfile.yml

% task
task: [default] task -l --sort none
task: Available tasks for this project:
* default:              show commands
* setup_snapshot:       [For Assignee] [After branch switch] Used when making changes to Snapshot Testing after switching from the develop branch.
(Example) task setup_snapshot or sh setup_snapshot.sh
* upload_snapshot:       [For Assignee] [During PR creation] Upload the snapshot images to the S3 bucket for PR review by passing the modified tests as arguments
(Example) task upload_snapshot -- ×××Tests or sh upload_snapshot.sh ×××Tests
* Download_snapshot:       [For Reviewer] [During review] Download the reference images by passing the relevant tests as arguments
(Example) task download_snapshot -- ×××Tests or sh download_snapshot.sh ×××Tests
* force_upload_snapshot:       [For Assignee] [After merging] If changes affect reference images other than the modified test files, (for example, when common components are updated), manually upload the changes by passing the modified tests as arguments.
(Example) task force_upload_snapshot -- ×××Tests or sh force_upload_snapshot.sh ×××Tests

Additionally, this is something I have set up personally, but I find it convenient to have an alias that changes the hardcoded profile name in the shell to the profile configured in your environment. (For those who prefer their own profile names) In this case, the profile hardcoded as user is changed to myroute-user.

alias sett="gsed -i 's/user/myroute-user/' setup_snapshot.sh && gsed -i 's/user/myroute-user/' upload_snapshot.sh && gsed -i 's/user/myroute-user/' download_snapshot.sh && gsed -i 's/user/myroute-user/' force_upload_snapshot.sh"

Bitrise

In my route, we use Bitrise for CI.

When a PR that includes changes to Snapshot Testing is merged, Bitrise automatically detects these changes and copies the reference images from the feature folder to the develop folder.

This ensures that the snapshot tests always run correctly in all situations.

Detecting subtle differences in reference images

Sometimes, differences are too subtle to see with the naked eye, but snapshot tests will still detect them and report errors.

Can’t see anything (3_3)?

In such cases, using ImageMagick to overlay the images can help you spot the differences more easily. By running the following command:

convert Snapshot/refarence.png -color-matrix "6x3: 1 0 0 0 0 0.4  0 1 0 0 0 0  0 0 1 0 0 0" ~/changeColor.png \
  && magick Snapshot/failure.png ~/changeColor.png -compose dissolve -define compose:args='60,100' -composite ~/Desktop/blend.png \
  && rm ~/changeColor.png

Blended image

You can see the overlaid images. Changing the hue of the reference image to a reddish tint before overlaying can make it easier to spot differences.

For added convenience, I recommend adding this command to your bashrc.

~/.bashrc

compare() {
  convert $1 -color-matrix "6x3: 1 0 0 0 0 0.4  0 1 0 0 0 0  0 0 1 0 0 0" ~/Desktop/changeColor.png;
  magick $1 ~/Desktop/changeColor.png -compose dissolve -define compose:args='60,100' -composite ~/Desktop/blend.png;
  rm ~/Desktop/changeColor.png
}

If the files are generally placed in the same location, you may only need to pass the test name as an argument instead of the entire path. Additionally, since images hosted online can also be processed, this method can be useful during reviews.

To wrap things up, I bring Surprise Interviews!

I interviewed my colleagues to get feedback on the implementation of Snapshot Test!

Chang-san said: "Thanks to Hosaka-san’s initial research, we are now able to handle snapshots in a more convenient way. With the help of Ryomm-san, various implementation methods were organized into documents to ensure we didn’t forget anything. It has been really great, and I am very greatful🙇‍♂️. Hosaka-san said: “The biggest bottleneck is the time it takes to run full tests, so I would like to work on reducing that in the future."

As for myself, I’ve noticed the frustration of having to fix Snapshot Tests when the logic changes but the screen remains unaffected. However, it’s been helpful to confirm that there were no differences when transitioning to SwiftUI, which I think was good!