KINTO Tech Blog
Operation

KINTO ONE Operation: Preparations for the Launch of the New Prius

Cover Image for KINTO ONE Operation: Preparations for the Launch of the New Prius

Introduction

Hello, this is Adegawa from KINTO Technologies New Vehicle Subscription Development Group.

As an application engineer, I have been in charge of the development, maintenance, and the operation of KINTO ONE, a Toyota New Vehicle subscription service in Japan for three years.

I love cars, so I am very happy that my work allows me to be part of it. At the moment, I am prioritizing my family, so I am driving a minivan, but I would like to ride a Toyota GR someday! My favorite engine mechanism is the cam gear train!

Let me share today a couple of projects that I've found particularly fulfilling and accomplished.

Preparing for the launch of the new Prius

January 10 2023 marked the new Prius (5th generation) launch. Prius is a model that attracts attention even within Toyota, which has established itself as an eco-friendly vehicle at the forefront of hybrid vehicles.

However, the Prius (predecessor) at the time the project was launched was not particularly popular among the KINTO ONE models, partly because it had already been released for a long time. So, personally, I did not think that the response would be particularly large even if the new model came out.

But the goal was to withstand*, 990 PV/min! (This number is 10 times higher than the highest load ever.)

At that time, we were only about 3% of our target.

There are three main countermeasures.
task

Countermeasure 1: adjusting the connection endpoint of the database

RDB was configured with a writer instance and a reader instance. However, due to the access of the reference system being overwhelmingly large from the front, the load was biased on the reader instance, and the writer one had more room. We tentatively changed the setting to connect to the writer instance regardless of reference or update destination from the front.

As a result, the load was balanced as intended. Since it was easier than adding a reader instance, I adopted this method first.

Countermeasure 2: quotation screen improvement

The next most visited screen after the top screen is the quotation screen for each car model. Here, we select car grade, package, option, etc., and we obtained the vehicle master information from the relational database table. This has been optimized to retrieve from the vehicle master information stored in the in-memory key/value cache (redis in Elasticache) to reduce DB load and improve front response.

Next, the information that could not be cached was used to identify the high DB load. From quotation to application, I consult the master table containing dealer store information along with vehicle details. While no issues arise in terms of slow queries, there is a notable volume of transactions, resulting in cumulative effects. To address this, I introduced an index to the specific high-cost target table, leading to a significant improvement in query performance, with costs now reduced to single digits.

sim-step

Countermeasure 3: infrastructure enhancement

In addition to the above measures, we made adjustments by reviewing the infrastructure structure so that we can withstand the target values of 1and 2. As a special temporary response, we increased the infrastructure as follows by manual scaling.

Scale out

AWS service Quantity - before change Quantity - after change
EC2 Front End 6 36
EC2 Back End 2 15

Scale up

AWS service Type - before Change Type - changed to
EC2 Scale out only
Elasticache t3.medium r6g.large
RDS r5.xlarge r5.12xlarge

Miscellaneous

To proactively handle a sudden surge in access, it's crucial to pre-warm the ALB (Application Load Balancer). We submitted a request to AWS, providing the following information for the pre-warming operations:

  • Expected peak hours
  • The number of requests expected during the peak hour
  • Traffic pattern
  • Use cases
  • etc.

On the day of release

As for the load, the actual access was about 10% at the peak compared to the allowable range that was prepared, but it was stable from the day of release without any trouble. The response from customers was good, and the influx (quotation and application) of the entire Prius, including the KINTO exclusive grade, seems to have been as expected. The part that had been tentatively responding after watching the transition for a while was calm was restored and finished.
access

Finally

It was a rare big event, so I think it was a very valuable experience. Whenever I saw a Prius in the city, I was curious and couldn’t help but follow them with my eyes. Could this be a KINTO one?

Looking at the results, it seems that the specifications of the infrastructure were excessive, but I think it was good because it was impossible to accurately predict the inflow in advance and there was no room for mistakes.

Despite the busy period from the end of the year to the beginning of the new year, I think that I was able to overcome it safely thanks to the cooperation of many colleagues.

Service providers who managed the load test plan and schedule. A team member from the oncoming system who took care of the service outage due to server expansion. We received advice from Platform G team members regarding infrastructure expansion.

I also want to express my gratitude here. Thank you very much.

Facebook

関連記事 | Related Posts

We are hiring!

【PjM】プロジェクト推進G/東京

プロジェクト推進グループについてプロジェクト推進グループでは、​TOYOTAのクルマのサブスクリプションサービスである『 KINTO ONE 』をはじめ、国内向けサービスのプロジェクト立ち上げから運用保守に至るまでの運営管理を行っています。

【フロントエンドエンジニア】新車サブスク開発G/東京

新車サブスク開発グループについてTOYOTAのクルマのサブスクリプションサービスである『 KINTO ONE 』のWebサイトの開発、運用をしています。​業務内容トヨタグループの金融、モビリティサービスの内製開発組織である同社にて、自社サービスである、TOYOTAのクルマのサブスクリプションサービス『KINTO ONE』のWebサイトの開発、運用を行っていただきます。