KINTO Tech Blog
Development

Schema-first Development with Protocol Buffers, GraphQL Schema, and Swagger Spec

Cover Image for Schema-first Development with Protocol Buffers, GraphQL Schema, and Swagger Spec

Introduction

I am Aoi Nakanishi, lead engineer of KINTO FACTORY at KINTO Technologies. The KINTO FACTORY project is redesigning the system with a view to service growth of supported vehicle models and products, as well as nationwide expansion. This project also incorporates with modern technologies and development workflows.

In this article, I will describe the schema-first development we are working on at KINTO FACTORY.

What is schema-first development?

This method, which involves defining a schema file, generating code using a code generator, and developing an API, solves the following problems.

  1. When trying to combine, it doesn't work due to type difference.
  2. The documents are outdated and the code is correct.
  3. Client implementation is duplicated for each language.

1. When trying to combine, it doesn't work due to type difference.

Since the schema is defined as an interface with the front end, back end, various microservices and external services, etc., discrepancies in data structures are less likely to occur.

2. The documents are outdated and the code is correct.

By using a generator to output documents, it is possible to avoid situations where the contents of the documents and the code tend to diverge as operations continue.

3. Client implementation is duplicated for each language.

Code is automatically generated from the defined schema file regardless of the development language the client is using on the web app or mobile app, etc., so it is possible to avoid unnecessary development work when implementing the same function in different languages, for example.

Other

Many people feel that the barrier to implementation is high if there is no one in the team with experience doing so. However, schema-first development provides a range of benefits for developers such as value validation, automatic code generation for mock servers, git version control, etc.

KINTO FACTORY system configuration

KINTO FACTORY uses the microservice architecture shown below.

  • GraphQL from browser
  • REST API from third-party system
  • gRPC (Protocol Buffers) between each microservice

These are the kinds of configurations it uses to communicate.

KINTO FACTORY SYSTEM DESIGN

Interface Description Language (IDL)

In general, each API design is defined using the following IDL (Interface Description Language).

Interface IDL
GraphQL GraphQL Schema https://graphql.org/learn/schema/
REST API Swagger Spec https://swagger.io/specification/
gRPC Protocol Buffers https://developers.google.com/protocol-buffers

Learning multiple IDLs is expensive and inefficient.

Schema conversion tools

I thought, if each IDL can define names and types and generate code, surely we can convert between schemas? I looked into it and summarized the findings in the table below.

Before conversion/After conversion GraphQL Schema Swagger Spec Protocol Buffers
GraphQL Schema - ? ?
Swagger Spec openapi-to-graphql - openapi2proto
Protocol Buffers go-proto-gql protoc-gen-openapiv2 -
  • There is not much information on tools that convert based on GraphQL Schema.
  • Tools to convert based on Swagger Spec have not been maintained for a long time.
  • Tools to convert based on Protocol Buffers have more options and information than the ones mentioned above.

Based on the above findings, we chose to define using Protocol Buffers and convert to another Schema.

Source file (.proto)

Preparation 1

Get the files necessary to define the Rest API from https://github.com/googleapis/googleapis

  1. google/api/annotations.proto
  2. google/api/http.proto
  3. google/api/httpbody.proto

Preparation 2

Get the proto definition file required to define the GraphQL Schema from https://github.com/danielvladco/go-proto-gql

  1. protobuf/graphql.proto

Definition file (example.proto)

The following definition file was created for this article using an article from the Tech Blog as an example.

syntax = "proto3"; 
package com.kinto_technologies.blog; 
option go_package = "blog.kinto-technologies.com"; 
import "google/api/annotations.proto"; // Load file acquired in Preparation 1 
import "protobuf/graphql.proto"; // Import file acquired in Preparation 2

// Article 
message Article { 
	// Title 
	string title = 1; 
	// Author 
	string author = 2; 
	// Content 
	string content = 3; 
}

// Request 
message Request { 
	uint64 id = 1; 
}

// Result 
message Result { 
	uint64 id = 1; 
}

// Tech Blog Service
service TechBlog { 
	// Post Article 
	rpc PostArticle(Article) returns (Result) { 
		option (google.api.http) = { 
			post: "/post" 
		}; 
		option (danielvladco.protobuf.graphql.rpc) = { 
			type: MUTATION 
		}; 
	}

    // Get Article
    rpc GetArticle(Request) returns (Article) { 
	    option (google.api.http) = { 
			get: "/get/{id}" 
		};
		option (danielvladco.protobuf.graphql.rpc) = { 
			type: QUERY 
		}; 
	} 
} 

Convert .from proto to .graphql

Install go-proto-gql

Clone repository

git clone https://github.com/danielvladco/go-proto-gql.git

cd go-proto-gql 

Install Protoc plugins

cd ./protoc-gen-gql 
go install 

Convert from .proto to .graphql

protoc --gql_out=paths=source_relative:. -I=. example.proto 

Output file (.graphql)

""" 
Tech Blog Service

""" 
directive @TechBlog on FIELD_DEFINITION 
""" 
Article

""" 
type Article { 
	""" 
	Title
	
	"""
	title: String
	"""
	 Author
	
	"""
	author: String
	"""
	 Content
	
	"""
	content: String
} 
""" 
Article

""" 
input ArticleInput { 
	""" 
	Title
	
	"""
	title: String
	"""
	 Author
	
	"""
	author: String
	"""
	 Content
	
	"""
	content: String
} 
type Mutation { 
	""" 
	Post Article
	
	"""
	techBlogPostArticle(in: ArticleInput): Result
} 
type Query { 
	""" 
	Get Article
	
	"""
	techBlogGetArticle(in: RequestInput): Article
} 
""" 
Request

""" 
input RequestInput { 
	id: Int 
} 
"""
 Result

""" 
type Result { 
	id: Int 
} 

Convert from .proto to .swagger.json

Install protobuf

brew install protobuf 

Install protocol-gen-openapiv2

go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@latest 

Convert from .proto to .swagger.json

protoc -I . --openapiv2_out=allow_merge=true,merge_file_name=./example:. example.proto 

Output file (.swagger.json)

{
   "swagger":"2.0",
   "info":{
      "title":"example.proto",
      "version":"version not set"
   },
   "tags":[
      {
         "name":"TechBlog"
      }
   ],
   "consumes":[
      "application/json"
   ],
   "produces":[
      "application/json"
   ],
   "paths":{
      "/get/{id}":{
         "get":{
            "summary":"Get Article",
            "operationId":"TechBlog_GetArticle",
            "responses":{
               "200":{
                  "description":"A successful response.",
                  "schema":{
                     "$ref":"#/definitions/blogArticle"
                  }
               },
               "default":{
                  "description":"An unexpected error response.",
                  "schema":{
                     "$ref":"#/definitions/rpcStatus"
                  }
               }
            },
            "parameters":[
               {
                  "name":"id",
                  "in":"path",
                  "required":true,
                  "type":"string",
                  "format":"uint64"
               }
            ],
            "tags":[
               "TechBlog"
            ]
         }
      },
      "/post":{
         "post":{
            "summary":"Post Article",
            "operationId":"TechBlog_PostArticle",
            "responses":{
               "200":{
                  "description":"A successful response.",
                  "schema":{
                     "$ref":"#/definitions/blogResult"
                  }
               },
               "default":{
                  "description":"An unexpected error response.",
                  "schema":{
                     "$ref":"#/definitions/rpcStatus"
                  }
               }
            },
            "parameters":[
               {
                  "name":"title",
                  "description":"Title",
                  "in":"query",
                  "required":false,
                  "type":"string"
               },
               {
                  "name":"author",
                  "description":"Author",
                  "in":"query",
                  "required":false,
                  "type":"string"
               },
               {
                  "name":"content",
                  "description":"Content",
                  "in":"query",
                  "required":false,
                  "type":"string"
               }
            ],
            "tags":[
               "TechBlog"
            ]
         }
      }
   },
   "definitions":{
      "blogArticle":{
         "type":"object",
         "properties":{
            "title":{
               "type":"string",
               "title":"Title"
            },
            "Author":{
               "type":"string",
               "title":"Author"
            },
            "content":{
               "type":"string",
               "title":"Content"
            }
         },
         "title":"Article"
      },
      "blogResult":{
         "type":"object",
         "properties":{
            "id":{
               "type":"string",
               "format":"uint64"
            }
         },
         "title":"Result"
      },
      "protobufAny":{
         "type":"object",
         "properties":{
            "@type":{
               "type":"string"
            }
         },
         "additionalProperties":{
            
         }
      },
      "rpcStatus":{
         "type":"object",
         "properties":{
            "code":{
               "type":"integer",
               "format":"int32"
            },
            "message":{
               "type":"string"
            },
            "details":{
               "type":"array",
               "items":{
                  "$ref":"#/definitions/protobufAny"
               }
            }
         }
      }
   }
}

Summary

In this article we introduced schema-first development and tools for converting schema definitions as a way of minimizing multiple schema definitions.

We hope that it will be helpful for those who want to resolve the confusion of multiple definition languages, especially those who are considering converting Protocol Buffers definitions to GraphQL Schema and Swagger Spec.

I hope to publish other articles on document generation, automatic generation of validation processing and automatic code generation, etc.

Follow us!

KINTO Technologies is now on Twitter. Please follow us to keep up with the latest information.

https://twitter.com/KintoTech_Dev

We are hiring!

KINTO Technologies is looking for people to work with us to create the future of mobility together. We also conduct informal interviews, so please feel free to contact us if you are interested.

https://www.kinto-technologies.com/recruit/

Facebook

関連記事 | Related Posts

Aoi Nakanishi
Aoi Nakanishi
Cover Image for Protocol Buffers, GraphQL Schema, Swagger Specで始めるスキーマファースト開発入門

Protocol Buffers, GraphQL Schema, Swagger Specで始めるスキーマファースト開発入門

Kang Juyoung
Kang Juyoung
Cover Image for Build Your Front-end with Atomic Design!

Build Your Front-end with Atomic Design!

Maya.S
Maya.S
Cover Image for October Welcomes: Introducing the New Members

October Welcomes: Introducing the New Members

Rina.K
Rina.K
Cover Image for Starting the KINTO Technologies Tech Blog

Starting the KINTO Technologies Tech Blog

M.Nishida
M.Nishida
Cover Image for Half-Year Anniversary of the Launch of KINTO FACTORY: A Path of Challenge and Learning

Half-Year Anniversary of the Launch of KINTO FACTORY: A Path of Challenge and Learning

S.Ito
S.Ito
Cover Image for Improving the Master Data Management of KINTO FACTORY

Improving the Master Data Management of KINTO FACTORY

We are hiring!

【KINTO FACTORYバックエンドエンジニア】プロジェクト推進G/大阪

KINTO FACTORYについて自動車のソフトウェア、ハードウェア両面でのアップグレードを行う新サービスです。トヨタ・レクサスの車をお持ちのお客様にOTAやハードウェアアップデートを通してリフォーム、アップグレード、パーソナライズなどを提供し購入後にも進化続ける自動車を提供するモビリティ業界における先端のサービスの開発となります。

【KINTO FACTORYバックエンドエンジニア(リーダークラス)】プロジェクト推進G/東京・大阪

KINTO FACTORYについて自動車のソフトウェア、ハードウェア両面でのアップグレードを行う新サービスです。トヨタ・レクサスの車をお持ちのお客様にOTAやハードウェアアップデートを通してリフォーム、アップグレード、パーソナライズなどを提供し購入後にも進化続ける自動車を提供するモビリティ業界における先端のサービスの開発となります。