Schema-first Development with Protocol Buffers, GraphQL Schema, and Swagger Spec
Introduction
I am Aoi Nakanishi, lead engineer of KINTO FACTORY at KINTO Technologies. The KINTO FACTORY project is redesigning the system with a view to service growth of supported vehicle models and products, as well as nationwide expansion. This project also incorporates with modern technologies and development workflows.
In this article, I will describe the schema-first development we are working on at KINTO FACTORY.
What is schema-first development?
This method, which involves defining a schema file, generating code using a code generator, and developing an API, solves the following problems.
- When trying to combine, it doesn't work due to type difference.
- The documents are outdated and the code is correct.
- Client implementation is duplicated for each language.
1. When trying to combine, it doesn't work due to type difference.
Since the schema is defined as an interface with the front end, back end, various microservices and external services, etc., discrepancies in data structures are less likely to occur.
2. The documents are outdated and the code is correct.
By using a generator to output documents, it is possible to avoid situations where the contents of the documents and the code tend to diverge as operations continue.
3. Client implementation is duplicated for each language.
Code is automatically generated from the defined schema file regardless of the development language the client is using on the web app or mobile app, etc., so it is possible to avoid unnecessary development work when implementing the same function in different languages, for example.
Other
Many people feel that the barrier to implementation is high if there is no one in the team with experience doing so. However, schema-first development provides a range of benefits for developers such as value validation, automatic code generation for mock servers, git version control, etc.
KINTO FACTORY system configuration
KINTO FACTORY uses the microservice architecture shown below.
- GraphQL from browser
- REST API from third-party system
- gRPC (Protocol Buffers) between each microservice
These are the kinds of configurations it uses to communicate.
Interface Description Language (IDL)
In general, each API design is defined using the following IDL (Interface Description Language).
| Interface | IDL | |
|---|---|---|
| GraphQL | GraphQL Schema | https://graphql.org/learn/schema/ |
| REST API | Swagger Spec | https://swagger.io/specification/ |
| gRPC | Protocol Buffers | https://developers.google.com/protocol-buffers |
Learning multiple IDLs is expensive and inefficient.
Schema conversion tools
I thought, if each IDL can define names and types and generate code, surely we can convert between schemas? I looked into it and summarized the findings in the table below.
| Before conversion/After conversion | GraphQL Schema | Swagger Spec | Protocol Buffers |
|---|---|---|---|
| GraphQL Schema | - | ? | ? |
| Swagger Spec | openapi-to-graphql | - | openapi2proto |
| Protocol Buffers | go-proto-gql | protoc-gen-openapiv2 | - |
- There is not much information on tools that convert based on GraphQL Schema.
- Tools to convert based on Swagger Spec have not been maintained for a long time.
- Tools to convert based on Protocol Buffers have more options and information than the ones mentioned above.
Based on the above findings, we chose to define using Protocol Buffers and convert to another Schema.
Source file (.proto)
Preparation 1
Get the files necessary to define the Rest API from https://github.com/googleapis/googleapis
Preparation 2
Get the proto definition file required to define the GraphQL Schema from https://github.com/danielvladco/go-proto-gql
Definition file (example.proto)
The following definition file was created for this article using an article from the Tech Blog as an example.
syntax = "proto3";
package com.kinto_technologies.blog;
option go_package = "blog.kinto-technologies.com";
import "google/api/annotations.proto"; // Load file acquired in Preparation 1
import "protobuf/graphql.proto"; // Import file acquired in Preparation 2
// Article
message Article {
// Title
string title = 1;
// Author
string author = 2;
// Content
string content = 3;
}
// Request
message Request {
uint64 id = 1;
}
// Result
message Result {
uint64 id = 1;
}
// Tech Blog Service
service TechBlog {
// Post Article
rpc PostArticle(Article) returns (Result) {
option (google.api.http) = {
post: "/post"
};
option (danielvladco.protobuf.graphql.rpc) = {
type: MUTATION
};
}
// Get Article
rpc GetArticle(Request) returns (Article) {
option (google.api.http) = {
get: "/get/{id}"
};
option (danielvladco.protobuf.graphql.rpc) = {
type: QUERY
};
}
}
Convert .from proto to .graphql
Install go-proto-gql
Clone repository
git clone https://github.com/danielvladco/go-proto-gql.git
cd go-proto-gql
Install Protoc plugins
cd ./protoc-gen-gql
go install
Convert from .proto to .graphql
protoc --gql_out=paths=source_relative:. -I=. example.proto
Output file (.graphql)
"""
Tech Blog Service
"""
directive @TechBlog on FIELD_DEFINITION
"""
Article
"""
type Article {
"""
Title
"""
title: String
"""
Author
"""
author: String
"""
Content
"""
content: String
}
"""
Article
"""
input ArticleInput {
"""
Title
"""
title: String
"""
Author
"""
author: String
"""
Content
"""
content: String
}
type Mutation {
"""
Post Article
"""
techBlogPostArticle(in: ArticleInput): Result
}
type Query {
"""
Get Article
"""
techBlogGetArticle(in: RequestInput): Article
}
"""
Request
"""
input RequestInput {
id: Int
}
"""
Result
"""
type Result {
id: Int
}
Convert from .proto to .swagger.json
Install protobuf
brew install protobuf
Install protocol-gen-openapiv2
go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@latest
Convert from .proto to .swagger.json
protoc -I . --openapiv2_out=allow_merge=true,merge_file_name=./example:. example.proto
Output file (.swagger.json)
{
"swagger":"2.0",
"info":{
"title":"example.proto",
"version":"version not set"
},
"tags":[
{
"name":"TechBlog"
}
],
"consumes":[
"application/json"
],
"produces":[
"application/json"
],
"paths":{
"/get/{id}":{
"get":{
"summary":"Get Article",
"operationId":"TechBlog_GetArticle",
"responses":{
"200":{
"description":"A successful response.",
"schema":{
"$ref":"#/definitions/blogArticle"
}
},
"default":{
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/rpcStatus"
}
}
},
"parameters":[
{
"name":"id",
"in":"path",
"required":true,
"type":"string",
"format":"uint64"
}
],
"tags":[
"TechBlog"
]
}
},
"/post":{
"post":{
"summary":"Post Article",
"operationId":"TechBlog_PostArticle",
"responses":{
"200":{
"description":"A successful response.",
"schema":{
"$ref":"#/definitions/blogResult"
}
},
"default":{
"description":"An unexpected error response.",
"schema":{
"$ref":"#/definitions/rpcStatus"
}
}
},
"parameters":[
{
"name":"title",
"description":"Title",
"in":"query",
"required":false,
"type":"string"
},
{
"name":"author",
"description":"Author",
"in":"query",
"required":false,
"type":"string"
},
{
"name":"content",
"description":"Content",
"in":"query",
"required":false,
"type":"string"
}
],
"tags":[
"TechBlog"
]
}
}
},
"definitions":{
"blogArticle":{
"type":"object",
"properties":{
"title":{
"type":"string",
"title":"Title"
},
"Author":{
"type":"string",
"title":"Author"
},
"content":{
"type":"string",
"title":"Content"
}
},
"title":"Article"
},
"blogResult":{
"type":"object",
"properties":{
"id":{
"type":"string",
"format":"uint64"
}
},
"title":"Result"
},
"protobufAny":{
"type":"object",
"properties":{
"@type":{
"type":"string"
}
},
"additionalProperties":{
}
},
"rpcStatus":{
"type":"object",
"properties":{
"code":{
"type":"integer",
"format":"int32"
},
"message":{
"type":"string"
},
"details":{
"type":"array",
"items":{
"$ref":"#/definitions/protobufAny"
}
}
}
}
}
}
Summary
In this article we introduced schema-first development and tools for converting schema definitions as a way of minimizing multiple schema definitions.
We hope that it will be helpful for those who want to resolve the confusion of multiple definition languages, especially those who are considering converting Protocol Buffers definitions to GraphQL Schema and Swagger Spec.
I hope to publish other articles on document generation, automatic generation of validation processing and automatic code generation, etc.
Follow us!
KINTO Technologies is now on Twitter. Please follow us to keep up with the latest information.
https://twitter.com/KintoTech_Dev
We are hiring!
KINTO Technologies is looking for people to work with us to create the future of mobility together. We also conduct informal interviews, so please feel free to contact us if you are interested.
関連記事 | Related Posts
We are hiring!
【フロントエンドエンジニア】プロジェクト推進G／東京
配属グループについて▶新サービス開発部 プロジェクト推進グループ 中古車サブスク開発チームTOYOTAのクルマのサブスクリプションサービスである『 KINTO ONE 中古車 』のWebサイトの開発、運用を中心に、その他サービスの開発、運用も行っています。
【PdM】共通サービス開発G／東京
共通サービス開発グループについてWebサービスやモバイルアプリの開発において、必要となる共通機能＝会員プラットフォームや決済プラットフォームなどの企画・開発を手がけるグループです。KINTOの名前が付くサービスやKINTOに関わりのあるサービスを同一のユーザーアカウントに対して提供し、より良いユーザー体験を実現できるよう、様々な共通機能や顧客基盤を構築していくことを目的としています。
イベント情報
