Schema-first Development with Protocol Buffers, GraphQL Schema, and Swagger Spec
Aoi Nakanishi
Introduction
I am Aoi Nakanishi, lead engineer of KINTO FACTORY at KINTO Technologies. The KINTO FACTORY project is redesigning the system with a view to service growth of supported vehicle models and products, as well as nationwide expansion. This project also incorporates with modern technologies and development workflows.
In this article, I will describe the schema-first development we are working on at KINTO FACTORY.
What is schema-first development?
This method, which involves defining a schema file, generating code using a code generator, and developing an API, solves the following problems.
- When trying to combine, it doesn't work due to type difference.
- The documents are outdated and the code is correct.
- Client implementation is duplicated for each language.
1. When trying to combine, it doesn't work due to type difference.
Since the schema is defined as an interface with the front end, back end, various microservices and external services, etc., discrepancies in data structures are less likely to occur.
2. The documents are outdated and the code is correct.
By using a generator to output documents, it is possible to avoid situations where the contents of the documents and the code tend to diverge as operations continue.
3. Client implementation is duplicated for each language.
Code is automatically generated from the defined schema file regardless of the development language the client is using on the web app or mobile app, etc., so it is possible to avoid unnecessary development work when implementing the same function in different languages, for example.
Other
Many people feel that the barrier to implementation is high if there is no one in the team with experience doing so. However, schema-first development provides a range of benefits for developers such as value validation, automatic code generation for mock servers, git version control, etc.
KINTO FACTORY system configuration
KINTO FACTORY uses the microservice architecture shown below.
- GraphQL from browser
- REST API from third-party system
- gRPC (Protocol Buffers) between each microservice
These are the kinds of configurations it uses to communicate.
Interface Description Language (IDL)
In general, each API design is defined using the following IDL (Interface Description Language).
| Interface | IDL | |
|---|---|---|
| GraphQL | GraphQL Schema | https://graphql.org/learn/schema/ |
| REST API | Swagger Spec | https://swagger.io/specification/ |
| gRPC | Protocol Buffers | https://developers.google.com/protocol-buffers |
Learning multiple IDLs is expensive and inefficient.
Schema conversion tools
I thought, if each IDL can define names and types and generate code, surely we can convert between schemas? I looked into it and summarized the findings in the table below.
| Before conversion/After conversion | GraphQL Schema | Swagger Spec | Protocol Buffers |
|---|---|---|---|
| GraphQL Schema | - | ? | ? |
| Swagger Spec | openapi-to-graphql | - | openapi2proto |
| Protocol Buffers | go-proto-gql | protoc-gen-openapiv2 | - |
- There is not much information on tools that convert based on GraphQL Schema.
- Tools to convert based on Swagger Spec have not been maintained for a long time.
- Tools to convert based on Protocol Buffers have more options and information than the ones mentioned above.
Based on the above findings, we chose to define using Protocol Buffers and convert to another Schema.
Source file (.proto)
Preparation 1
Get the files necessary to define the Rest API from https://github.com/googleapis/googleapis
Preparation 2
Get the proto definition file required to define the GraphQL Schema from https://github.com/danielvladco/go-proto-gql
Definition file (example.proto)
The following definition file was created for this article using an article from the Tech Blog as an example.
syntax = "proto3";
package com.kinto_technologies.blog;
option go_package = "blog.kinto-technologies.com";
import "google/api/annotations.proto"; // Load file acquired in Preparation 1
import "protobuf/graphql.proto"; // Import file acquired in Preparation 2
// Article
message Article {
// Title
string title = 1;
// Author
string author = 2;
// Content
string content = 3;
}
// Request
message Request {
uint64 id = 1;
}
// Result
message Result {
uint64 id = 1;
}
// Tech Blog Service
service TechBlog {
// Post Article
rpc PostArticle(Article) returns (Result) {
option (google.api.http) = {
post: "/post"
};
option (danielvladco.protobuf.graphql.rpc) = {
type: MUTATION
};
}
// Get Article
rpc GetArticle(Request) returns (Article) {
option (google.api.http) = {
get: "/get/{id}"
};
option (danielvladco.protobuf.graphql.rpc) = {
type: QUERY
};
}
}
Convert .from proto to .graphql
Install go-proto-gql
Clone repository
git clone https://github.com/danielvladco/go-proto-gql.git
cd go-proto-gql
Install Protoc plugins
cd ./protoc-gen-gql
go install
Convert from .proto to .graphql
protoc --gql_out=paths=source_relative:. -I=. example.proto
Output file (.graphql)
"""
Tech Blog Service
"""
directive @TechBlog on FIELD_DEFINITION
"""
Article
"""
type Article {
"""
Title
"""
title: String
"""
Author
"""
author: String
"""
Content
"""
content: String
}
"""
Article
"""
input ArticleInput {
"""
Title
"""
title: String
"""
Author
"""
author: String
"""
Content
"""
content: String
}
type Mutation {
"""
Post Article
"""
techBlogPostArticle(in: ArticleInput): Result
}
type Query {
"""
Get Article
"""
techBlogGetArticle(in: RequestInput): Article
}
"""
Request
"""
input RequestInput {
id: Int
}
"""
Result
"""
type Result {
id: Int
}
Convert from .proto to .swagger.json
Install protobuf
brew install protobuf
Install protocol-gen-openapiv2
go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@latest
Convert from .proto to .swagger.json
protoc -I . --openapiv2_out=allow_merge=true,merge_file_name=./example:. example.proto
Output file (.swagger.json)
{
"swagger":"2.0",
"info":{
"title":"example.proto",
"version":"version not set"
},
"tags":[
{
"name":"TechBlog"
}
],
"consumes":[
"application/json"
],
"produces":[
"application/json"
],
"paths":{
"/get/{id}":{
"get":{
"summary":"Get Article",
"operationId":"TechBlog_GetArticle",
"responses":{
"200":{
"description":"A successful response.",
"schema":{
"$ref":"#/definitions/blogArticle"
}
},
"default":{
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/rpcStatus"
}
}
},
"parameters":[
{
"name":"id",
"in":"path",
"required":true,
"type":"string",
"format":"uint64"
}
],
"tags":[
"TechBlog"
]
}
},
"/post":{
"post":{
"summary":"Post Article",
"operationId":"TechBlog_PostArticle",
"responses":{
"200":{
"description":"A successful response.",
"schema":{
"$ref":"#/definitions/blogResult"
}
},
"default":{
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/rpcStatus"
}
}
},
"parameters":[
{
"name":"title",
"description":"Title",
"in":"query",
"required":false,
"type":"string"
},
{
"name":"author",
"description":"Author",
"in":"query",
"required":false,
"type":"string"
},
{
"name":"content",
"description":"Content",
"in":"query",
"required":false,
"type":"string"
}
],
"tags":[
"TechBlog"
]
}
}
},
"definitions":{
"blogArticle":{
"type":"object",
"properties":{
"title":{
"type":"string",
"title":"Title"
},
"Author":{
"type":"string",
"title":"Author"
},
"content":{
"type":"string",
"title":"Content"
}
},
"title":"Article"
},
"blogResult":{
"type":"object",
"properties":{
"id":{
"type":"string",
"format":"uint64"
}
},
"title":"Result"
},
"protobufAny":{
"type":"object",
"properties":{
"@type":{
"type":"string"
}
},
"additionalProperties":{
}
},
"rpcStatus":{
"type":"object",
"properties":{
"code":{
"type":"integer",
"format":"int32"
},
"message":{
"type":"string"
},
"details":{
"type":"array",
"items":{
"$ref":"#/definitions/protobufAny"
}
}
}
}
}
}
Summary
In this article we introduced schema-first development and tools for converting schema definitions as a way of minimizing multiple schema definitions.
We hope that it will be helpful for those who want to resolve the confusion of multiple definition languages, especially those who are considering converting Protocol Buffers definitions to GraphQL Schema and Swagger Spec.
I hope to publish other articles on document generation, automatic generation of validation processing and automatic code generation, etc.
Follow us!
KINTO Technologies is now on Twitter. Please follow us to keep up with the latest information.
https://twitter.com/KintoTech_Dev
We are hiring!
KINTO Technologies is looking for people to work with us to create the future of mobility together. We also conduct informal interviews, so please feel free to contact us if you are interested.
関連記事 | Related Posts
Aoi Nakanishi
Development
Protocol Buffers, GraphQL Schema, Swagger Specで始めるスキーマファースト開発入門
maya.s
General
October Welcomes: Introducing the New Members
Rina.K
General
Starting the KINTO Technologies Tech Blog
M.Nishida
Operation
Half-Year Anniversary of the Launch of KINTO FACTORY: A Path of Challenge and Learning
radegawa
Operation
KINTO ONE Operation: Preparations for the Launch of the New Prius
S.Ito
GeneralImproving the Master Data Management of KINTO FACTORY
We are hiring!
【KINTO FACTORYフルスタックエンジニア】プロジェクト推進G／東京・大阪
KINTO FACTORYについて自動車のソフトウェア、ハードウェア両面でのアップグレードを行う新サービスです。トヨタ・レクサスの車をお持ちのお客様にOTAやハードウェアアップデートを通してリフォーム、アップグレード、パーソナライズなどを提供し購入後にも進化続ける自動車を提供するモビリティ業界における先端のサービスの開発となります。
【KINTO FACTORYバックエンドエンジニア】プロジェクト推進G／東京・大阪
KINTO FACTORYについて自動車のソフトウェア、ハードウェア両面でのアップグレードを行う新サービスです。トヨタ・レクサスの車をお持ちのお客様にOTAやハードウェアアップデートを通してリフォーム、アップグレード、パーソナライズなどを提供し購入後にも進化続ける自動車を提供するモビリティ業界における先端のサービスの開発となります。