Introduction

Hello. My name is Shiode, and I'm in charge of payment-related back-end development at the Woven Payment Solution Group.

Our group uses Kotlin for development, and we use Ktor as the web framework. Ktor requires a separate JSON library to process JSON request bodies and return JSON responses. At the beginning, we used Gson as a JSON library, but we encountered problems using Gson, so we switched to Moshi.

Today, I'm going to describe why we chose Moshi, the problems we faced when we used Moshi with Ktor, and how we solved them.

The version of Ktor used in this article is the 2.x series.

Switching from Gson to Moshi

Issues with Gson

Initially, we were using Gson as a Ktor JSON library. However, when receiving request bodies, we discovered that problems would occur if a necessary key was missing.

For example, think about an endpoint that processes input as follows.

data class Input (
  val required: String,
  val optional: String? = null,
)

post("/") {
    val input = call.receive<Input>()
    call.respondText("required was ${input.required}", status = HttpStatusCode.OK)
}

Since String type is required for the Input class, null is not permitted. However, null is permitted with optional. This endpoint is a simple one that receives the above Input as a request body and returns the required of the received body.

With this endpoint, a NullPointerException (commonly known as Nullpo) will occur if you send an empty JSON {} that does not have a required. It would be preferable for an exception to occur during deserialization so as not to affect later processing, but an exception ends up occurring when accessing that variable, in this case call.respondText().

As a solution, there may be a way to make the Input class required a nullable type and perform a null check. However, since we're using Kotlin, we wanted to express it as a type where nulls are not allowed as far as possible, and write processing that is suitable for Kotlin.

Therefore, it became necessary to select a JSON library that would throw an exception during deserialization.

Why Did We Choose Moshi?

This project is developed with a schema base that uses OpenAPI, and the generation of interacting classes depends on the OpenAPI Generator. Also, since the client also uses the one generated by the OpenAPI Generator, a JSON library supported by OpenAPI was brought up as a replacement candidate.

The JSON libraries supported by OpenAPI Generator are shown below. Gson is excluded.

• Moshi
• Jackson
• kotlinx.serialization

Jackson can support null safe by enabling Option, but when we looked at the relevant source code, performance was described as being affected. We decided to eliminate it as a candidate without comparing performance.

Meanwhile, kotlinx.serialization is a library developed by JetBrains, and the fact that it's native to Kotlin and compatible with Ktor made it is a good candidate. However, Moshi has twice as many stars as kotlinx.serialization (at time of writing, Moshi: 8.5K, kotlinx.serialization: 4.1K), and Moshi is Gson v3, so we decided to adopt Moshi.

Problems Using Moshi with Ktor

According to Ktor's PR#2004, there is no Ktor support for Moshi at the time of writing. Also, according to PR#2005, there seems to be no policy to provide Moshi as a core feature of Ktor in the future, either.

On the other hand, it seems that Ktor is planning to provide a Marketplace, so there is a possibility that in the future a third party will provide a Ktor plugin that will allow Moshi to be used. But there is no support for it right now, so you'll have to make your own Ktor plugin to be able to use Moshi.

Note that there is no information yet about the Marketplace at the time of writing. We're looking forward to hearing more about it.

Solution

Create a Ktor Plug-in to Enable Use of Moshi

The following are the classes required for using Moshi with Ktor. This implementation is almost the same as GsonConverter for Ktor.

class MoshiConverter(
    private val moshi: Moshi = Moshi::Builder().build(),
) : ContentConverter {
    override suspend fun serialize(
        contentType: ContentType,
        charset: Charset,
        typeInfo: TypeInfo,
        value: Any 
    ): OutgoingContent? {
        return TextContent(
             moshi.adapter(value.javaClass).toJson(value),
             contentType.withCharsetIfNeeded(charset)
        )
    }

    override suspend fun deserialize(charset: Charset, typeInfo: TypeInfo, content: ByteReadChannel): Any? {
        return withContext(Dispatchers.IO) {
            val body = content.toInputStream().reader(charset).buffered().use { it.readText() }
            moshi.adapter(typeInfo.type.java).fromJson(body)
        }
    }
}

It is used as shown below.

install(ContentNegotiation) {
    register(ContentType.Application.Json, MoshiConverter())
}

Compatibility with Moshi ArrayList

When we use the above Ktor plugin, it deserializes as a JAVA type instead of a Kotlin type when deserializing. For example, it will deserialize as an ArrayList even though you want to store it in a List. Moshi doesn't support ArrayList deserialization by default, so an exception will be thrown when you try to deserialize a List. To support any type in Moshi, you need to prepare an Adapter for that type. Since we want to support ArrayList this time, we must prepare a Moshi adapter for ArrayList ourselves.

The following adapters support ArrayList. This is almost identical to Moshi's CollectionJsonAdapter implementation.

abstract class ArrayListJsonAdapter<C : MutableCollection<T?>, T> private constructor(
    private val elementAdapter: JsonAdapter<T>
) : JsonAdapter<C>() {
    abstract fun newCollection(): C

    override fun fromJson(reader: JsonReader): C {
        val result = newCollection()
        reader.beginArray()
        while (reader.hasNext()) {
            result.add(elementAdapter.fromJson(reader)!!)
        }
        reader.endArray()
        return result
    }

    override fun toJson(writer: JsonWriter, value: C?) {
        writer.beginArray()
        for (element in value!!) {
            elementAdapter.toJson(writer, element)
        }
        writer.endArray()
    }

    override fun toString(): String {
        return "$elementAdapter.collection()"
    }

    companion object Factory : JsonAdapter.Factory {
        override fun create(type: Type, annotations: Set<Annotation>, moshi: Moshi): JsonAdapter<*>? {
            if (annotations.isNotEmpty()) return null
            return when (type.rawType) {
                ArrayList::class.java -> {
                    newArrayListAdapter<Any>(type, moshi).nullSafe()
                }
                Set::class.java -> {
                    newLinkedHashSetAdapter<Any>(type, moshi).nullSafe()
                }
                else -> null
            }
        }

        private fun <T> newArrayListAdapter(type: Type, moshi: Moshi): JsonAdapter<MutableCollection<T?>> {
            val elementType = Types.collectionElementType(type, Collection::class.java)
            val elementAdapter = moshi.adapter<T>(elementType)
            return object : ArrayListJsonAdapter<MutableCollection<T?>, T>(elementAdapter) {
                override fun newCollection(): MutableCollection<T?> = ArrayList()
            }
        }

        private fun <T> newLinkedHashSetAdapter(type: Type, moshi: Moshi): JsonAdapter<MutableSet<T?>> {
            val elementType = Types.collectionElementType(type, Collection::class.java)
            val elementAdapter = moshi.adapter<T>(elementType)
            return object : ArrayListJsonAdapter<MutableSet<T?>, T>(elementAdapter) {
                override fun newCollection(): MutableSet<T?> = LinkedHashSet()
            }
        }
    }
} 

Verifying Operation

After creating a Ktor plug-in for using Moshi with Ktor and a Moshi adapter for deserializing ArrayList with Moshi, the next step is to verify operation. In order to use the one I just described, we need to write the following

install(ContentNegotiation) { 
    register(ContentType.Application.Json, MoshiConverter( moshi = Moshi::Builder().add(ArrayListJsonAdapter.Factory) 
    )) 
} 

Then, let's try sending an empty JSON {} to the endpoint introduced in the first example in this article. Here's that endpoint again.

data class Input ( 
    val required: String,
    val optional: String? = null, 
) 

post("/") { 
    val input = call.receive<Input>() 
    call.respondText("required was ${input.required}", status = HttpStatusCode.OK) 
} 

At first, nullpo was generated when accessing required, but now the following exception occurs for call.receive<Input>() during deserialization, as expected.

com.squareup.moshi.JsonDataException: Required value ‘required’ missing at $ 

Conclusions

• Nullpo can occur when using Gson with Ktor.
• If you don't want nullpo to occur, you should choose another JSON library.
• To use Moshi with Ktor, you need to create your own Ktor plugin by yourself.
  • If you do that, you need to make a Moshi adapter to deserialize ArrayList.

References

• "Gson is deprecated," so try Moshi
• How to serialize ArrayList<float[]> using Moshi JSON library for Android