Introduction

Hello, this is Adegawa from KINTO Technologies New Vehicle Subscription Development Group.

As an application engineer, I have been in charge of the development, maintenance, and the operation of KINTO ONE, a Toyota New Vehicle subscription service in Japan for three years.

I love cars, so I am very happy that my work allows me to be part of it. At the moment, I am prioritizing my family, so I am driving a minivan, but I would like to ride a Toyota GR someday! My favorite engine mechanism is the cam gear train!

Let me share today a couple of projects that I've found particularly fulfilling and accomplished.

Preparing for the launch of the new Prius

January 10 2023 marked the new Prius (5th generation) launch. Prius is a model that attracts attention even within Toyota, which has established itself as an eco-friendly vehicle at the forefront of hybrid vehicles.

However, the Prius (predecessor) at the time the project was launched was not particularly popular among the KINTO ONE models, partly because it had already been released for a long time. So, personally, I did not think that the response would be particularly large even if the new model came out.

But the goal was to withstand*, 990 PV/min! (This number is 10 times higher than the highest load ever.)

At that time, we were only about 3% of our target.

There are three main countermeasures.

Countermeasure 1: adjusting the connection endpoint of the database

RDB was configured with a writer instance and a reader instance. However, due to the access of the reference system being overwhelmingly large from the front, the load was biased on the reader instance, and the writer one had more room. We tentatively changed the setting to connect to the writer instance regardless of reference or update destination from the front.

As a result, the load was balanced as intended. Since it was easier than adding a reader instance, I adopted this method first.

Countermeasure 2: quotation screen improvement

The next most visited screen after the top screen is the quotation screen for each car model. Here, we select car grade, package, option, etc., and we obtained the vehicle master information from the relational database table. This has been optimized to retrieve from the vehicle master information stored in the in-memory key/value cache (redis in Elasticache) to reduce DB load and improve front response.

Next, the information that could not be cached was used to identify the high DB load. From quotation to application, I consult the master table containing dealer store information along with vehicle details. While no issues arise in terms of slow queries, there is a notable volume of transactions, resulting in cumulative effects. To address this, I introduced an index to the specific high-cost target table, leading to a significant improvement in query performance, with costs now reduced to single digits.

Countermeasure 3: infrastructure enhancement

In addition to the above measures, we made adjustments by reviewing the infrastructure structure so that we can withstand the target values of 1and 2. As a special temporary response, we increased the infrastructure as follows by manual scaling.

Scale out

AWS service	Quantity - before change	Quantity - after change
EC2 Front End	6	36
EC2 Back End	2	15

Scale up

AWS service	Type - before Change	Type - changed to
EC2	Scale out only
Elasticache	t3.medium	r6g.large
RDS	r5.xlarge	r5.12xlarge

Miscellaneous

To proactively handle a sudden surge in access, it's crucial to pre-warm the ALB (Application Load Balancer). We submitted a request to AWS, providing the following information for the pre-warming operations:

• Expected peak hours
• The number of requests expected during the peak hour
• Traffic pattern
• Use cases
• etc.

On the day of release

As for the load, the actual access was about 10% at the peak compared to the allowable range that was prepared, but it was stable from the day of release without any trouble. The response from customers was good, and the influx (quotation and application) of the entire Prius, including the KINTO exclusive grade, seems to have been as expected. The part that had been tentatively responding after watching the transition for a while was calm was restored and finished.

Finally

It was a rare big event, so I think it was a very valuable experience. Whenever I saw a Prius in the city, I was curious and couldn’t help but follow them with my eyes. Could this be a KINTO one?

Looking at the results, it seems that the specifications of the infrastructure were excessive, but I think it was good because it was impossible to accurately predict the inflow in advance and there was no room for mistakes.

Despite the busy period from the end of the year to the beginning of the new year, I think that I was able to overcome it safely thanks to the cooperation of many colleagues.

Service providers who managed the load test plan and schedule. A team member from the oncoming system who took care of the service outage due to server expansion. We received advice from Platform G team members regarding infrastructure expansion.

I also want to express my gratitude here. Thank you very much.