Reading Vehicle Inspection QR Codes

Introduction

Hello. Thank you for reading this! I am Nakamoto, and I work in frontend development for the KINTO FACTORY service (hereinafter FACTORY). Since I decided to participate on the Advent Calendar series, I will talk about how we implemented vehicle inspection QR code reading, which we released this year.

Motivations for Adoption

At FACTORY, we are currently developing a service that allows customers to ride their cars for a long time from the perspectives of "Upgrade," "Renovation," and "Personalization." FACTORY's frontend functions like an e-commerce site, allowing customers to search for products that can be installed based on their vehicle information and make applications on the Internet. Information on the vehicle they are currently driving, such as the model, model year, and grade, is important.

Based on that kind of information, the service determines which products can be installed and whether or not they can be combined. In order to accurately determine such combinations and possibilities, FACTORY uses something called a chassis number. Some of you may know this, but when you look at a vehicle inspection form, you can find a serial number that is a chassis number. (The third field from the top left in the figure below)

*Source: Ministry of Land, Infrastructure, Transport and Tourism "Vehicle Inspection Certificate (Vehicle Inspection Certificate Sample)" (https://wwwtb.mlit.go.jp/hokkaido/content/000176963.pdf)

With FACTORY, you can easily search for products that can be attached to your vehicle by entering its chassis number.

Product listing page (Products have been searched by chassis number)

Right after the release of FACTORY, the user had to enter the chassis number manually using the form in the figure below.

Chassis number entry form

We released the ability to search by entering a chassis number this June, but at first, I thought it would be difficult to enter unfamiliar alphanumeric characters directly into the input field. Most people access FACTORY on their smartphones, as is the case with many eCommerce websites nowadays.

Typing a long string of alphanumeric characters using a smartphone can be troublesome on a small screen, and can lead to input mistakes. When this happens, it will show a message saying that the vehicle is not compatible with FACTORY, making the user unable to find the products they are looking for. I believed this could be a potential factor contributing to missed opportunities.

Then I noticed the QR code in the bottom right of the vehicle inspection.

Information That Can Be Read Using a Vehicle Inspection QR Code

As indicated by the Ministry of Land, Infrastructure, Transport and Tourism's electronic vehicle inspection website, the chassis number used in the above search can be read using a vehicle inspection QR code.

In addition, if you find a product you want to attach on FACTORY and want to buy it, you must first register your vehicle information. A chassis number and license plate information are needed to register. The vehicle inspection QR code basically contains the information written in the car inspection. We thought that by reading these, it would be easy to search for products and register a vehicle when buying.

As you can see in the figure below, there are QR codes in groups or two or three, and it is necessary to have them read correctly in either case without failure. Each QR code has a separate piece of information, and they must be combined in correct order after all of the codes are read. We wanted it to give feedback on the screen so the user can see which code in each group is being read.

*Source: Ministry of Land, Infrastructure, Transport and Tourism "About QR Codes" (https://www.denshishakensho-portal.mlit.go.jp/assets/files/Two-dimensional_code_item_definition.pdf)

Implementing the Reading UI

After the codes are read with a smartphone camera, they go through image processing and are analyzed and decoded with a JavaScript library. It was our first time using a camera to read and decode QR codes, so we tested it while gathering information on the Internet. We chose to use the getUserMedia() web API to read with the camera, and the js library qr-scanner to analyze QR codes.

The code read by getUserMedia is copied to a canvas element, and transferred to qr-scanner, which analyzes the code and determines which code from the data was read.

When testing the implementation, reading was a little difficult, and the QR code reading rate was slightly low for new vehicle inspections. We switched to electronic vehicle inspections in January. People who bought a vehicle or did a vehicle inspection already know this, but the form is considerably smaller (size B5) than the previous one (size A4). Because of this, the QR codes became a little denser and difficult to read.

By looking at the qr-scanner source and adjusting the Grayscale settings, the new and old car inspection types can both be read with some accuracy. After the image data is copied to the canvas, it is converted to a black-and-white image using the above Grayscale settings and sent to the QR code analysis library. Depending on the values, the car inspection pattern was reflected as garbage, and that seemed to lead to a lower reading rate.

https://github.com/nimiq/qr-scanner/blob/master/src/worker.ts#L73-L79

When implementing the UI, we wanted to show the positions of the readable QR codes by group while looking at the images captured by the camera in real time, so we arranged the necessary number of boxes at the top and displayed a check icon on the border of readable QR codes.

The reading screen looked like the figure below. You can see that the positions of the codes are recognized correctly regardless of the order in which they are read. After all of the information is read, the screen transitions the same way as when you search for a chassis number manually.

After Release

We released the QR code search function in October. For a while after the release, it was used only for the product list page as shown in the example above, so it is hard to say that this function was used a lot, but it is now used for the home screen. By reading the QR code on the home screen, it is possible to check for compatible vehicles and product compatibility both at once, and we think it will be used more.

As I mentioned in the "readable information" part, the QR codes also contain information needed for vehicle registration, so we will use this information for vehicle registration in the future and aim to make it more convenient and reduce input errors by reducing manual inputs as much as possible.

Conclusion

The rise of smartphones, enrichment of web content, and expansion of various web APIs have expanded the possibilities of websites. We will actively try new technologies with user convenience in mind, such as real-time OCR, NFC tags, and other contactless technologies.

We will explore various ideas to make KINTO FACTORY easier and more fun to use!

Lastly, KINTO FACTORY is looking for people to work with. If you are interested, check out the job openings below!