Developing server-side with Kotlin & entering Clova Skill Awards

Hello fellow readers! We are, Kagaya, working on the LINE GAME Platform development team and Akira, on the LINE LIVE Android app development team. Have you ever used Kotlin? Kotlin is a static-typed programming language that runs on JVM, and has been developed by JetBrains, well-known for making IDEs like IntelliJ and Android Studio. Kotlin has been announced as an official development language for Android in Google I/O last year. We’ve shared that we are using Kotlin for our Android app in the post, Why we use Kotlin in LINE Creators Studio. We are still using Kotlin for our new apps and also, we’ve been converting existing apps into Kotlin, and are in the process of refactoring. Many of LINE Android apps, including LINE the message app, LINE LIVE, LINE Creators Studio and Clova (AI assistant) have been employing Kotlin.

Lately, we’ve been employing Kotlin on the server-side as well. I’d like to share some use cases and our know-how on using Kotlin on the server-side:

Some of the content we share may have some similarities to developing Android apps; if you are an Android developer, we’d like to recommend you to have a go at the server-side with Kotlin. It won’t be that difficult to get started.

Kotlin and LINE GAME Platform servers

Many of the games provided with the LINE GAME Platform are outsourced or are developed by the LINE GAME studios. To help the game developers concentrate more on the game logic, the LINE GAME Platform provides features for authentication, social graph, promotion, and communication (user targeting, notification, board/chatting within game). We had had a chance to introduce our work at LINE DEVELOPER DAY 2017. Being a platform, the LINE GAME Platform consists of many independent components, and we usually start from scratch for new components and we use Kotlin for some of them.

Let’s briefly run through the components that use Kotlin. One of the LINE GAME Platform features is adding LINE friends to the game who have never played the game for those who have connected their LINE account with the game. We’ve recently added a feature where we recommend friends who are more likely to play the game, using our own algorithm. The following is an illustration of our logic.

First, we fetch the recommendation configuration (1), get the player’s friend information (2), obtain their profiles (3), and then the Recommendation API server ranks the friends by the likelihood for playing the game. The result is returned to the player. Steps (1) to (3) are based on a network connection. We’ve used Kotlin and Spring Boot 1.5 for the Recommendation API server.

Why Kotlin?

Here are a few reasons why we’ve used Kotlin:

Java compatibility

Our team has constructed a Gradle multi-project in a somewhat monolithic design, and only some of the sub-projects are written in Kotlin. So, there are cases where libraries written in Java are used by Kotlin apps. Here is what the Kotlin’s official website states:

100% interoperable with Java™

Just as their statement, Kotlin is highly compatible with Java, so there is almost — will explain why “almost” later — no problem calling each other. Since Kotlin is still “young”, there aren’t many libraries (such as O/R mapper and HTTP clients) we can use out there. But not to worry, since we can use Java libraries, we can use Retrofit and MyBatis. Kotlin also provides Extension with which you can easily make existing libraries to support Kotlin. An example of this is jackson-module-kotlin, a Kotlin plug-in for Jackson, a JSON library.

Coroutines

Kotlin supports coroutines. Coroutines are tasks that implement co-operative multitasking and the implementation itself. Unlike ordinary threads, coroutines are not mapped to native threads and do not require context switching which in turn helps fast operation. They are handy in developing web applications, in handling asynchronous processes.

The API server in the previous diagram uses coroutines to parallelize the HTTP call((3) in the diagram above) which brings the player’s profile information from another service. Here is a simplified version of the code.

val ctx = newFixedThreadPoolContext(NUMBER_OF_TRHEADS, "user")

fun getFriends(userIds: List<String>): Map<String, UserProfile> = runBlocking {
    userIds.chunked(PROFILE_MAX_PER_REQUEST).map { subset ->
        async(ctx) {
            mapper.readValue<List<UserProfile>>(khttp.get(URL, params = mapOf("ids" to subset.joinToString(","))).text)
        }
    }.map {
        it.await()
    }.flatMap {
        it
    }.associate {
        it.userId to it
    }
}

Since the values returned by the APIs we call don’t change frequently, we can anticipate a high cache hit rate. Hence, we use khttp, a simple blocking HTTP client. We use our own thread-pool based dispatcher on CoroutineContext to send HTTP requests simultaneously, while controlling the number of requests sent at the same time. The API for returning profile information uses the endpoint that handles multiple profiles. We divide the requests by the maximum number of profile information the endpoint can handle, defined by PROFILE_MAX_PER_REQUEST.

The sample we’ve looked at is simple asynchronous code, but is it a good one? No, it’s not. Kotlin coroutines have this special feature where you can pause a function operating, to increase the efficiency of CPU usage. This is why coroutines are suitable for non-blocking I/O. Also, one of the purpose of coroutines is to make improvement in using callbacks — a common way used for asynchronous implementations — and future/promise based implementations, due to being able to write asynchronous code as if it’s synchronous. You can see the strength of coroutines more in handling complex processes that require remote calls. Let’s see it with code. We can write the code sample above like the following, by using Netty-based HTTP client, AsyncHttpClient.

val client = asyncHttpClient()
    
fun getFriends2(userIds: List<String>): Map<String, UserProfile> {
    return userIds.chunked(PROFILE_MAX_PER_REQUEST).map { subset ->
        asyncApiCall(subset.joinToString(","))
    }.map {
        runBlocking {
            it.await()
        }
    }.flatMap {
        it
    }.associate {
        it.userId to it
    }
}

fun asyncApiCall(idsStr: String) = async {
    val response = client.prepareGet(URL).addQueryParam("ids", idsStr).execute().await()
    mapper.readValue<List<UserProfile>>(response.responseBody)
}

suspend fun <T> ListenableFuture<T>.await(): T = suspendCoroutine { continuation ->
    toCompletableFuture().whenComplete { result, exception ->
        if (exception == null) {
            continuation.resume(result)
        } else {
            continuation.resumeWithException(exception)
        }
    }
}

This code is not casting a limit on the number of simultaneous requests, but you can set the limit by configuring AsyncHttpClient or using Kotlin’s Channels, that implements a pattern similar to a pub-sub (Publish-Subscribe) pattern. As you can see from the code above, integrating asynchronous communication library can be easily done. A number of libraries such as Rx, Reactor, nio, and Guava ListenableFuture are already officially integrated.

If you’d like to know the basics of coroutines, I recommend the official documentation and a presentation by Roman Elizarov of Jetbrains in 2017. For those who are interested in finding out how coroutines let you integrate with the existing callback-based and future-based libraries, check out another presentation of Elizarov, Deep Dive into Coroutines on JVM. I highly recommend this particular presentation to understand coroutine implementation and use it properly. Kotlin coroutines are still experimental, although Andrey Breslav from JetBrain and Elizarov claim that coroutines are production ready. This article suggests the reason why coroutines still experimental is because there is a possibility to change its architecture based on user feedback. Coroutines will reach stable status in Kotlin v1.3*. The kotlinx.coroutines.experimental is to stay after the final release, so we will be able to continue using it as it is for a while.

* This was so at the time of publishing its original content in Japanese, which was August 2018. Kotlin v1.3 has been released, at the time of publishing the English translation of this article.

What else?

Both server side and client side can take benefit of most of Kotlin’s strengths, including the ones mentioned in Why we use Kotlin in in LINE Creators Studio. Let me list a few:

  • Data class that is to write as a  POJO
    • There is no need for Lombok
    • Useful in writing DTO (Data Transfer Object) for developing an API server
  • Null safety
  • Easily assigning read-only variables, with val
  • Sealed class, an extension of enum
  • Operator overloading
  • and more

If you are a Java programmer, when you try coding in Kotlin, you will probably encounter several “wow!” moments. My “wow” moment was when I tried using try & catch and if statements that are used as expressions. For more information, check out the official documentation. Being an IDE developing company, JetBrains documentation is easy to understand.

Things you miss out often in developing the server side in Kotlin

Spring claims that Spring 5 and Spring Boot v2.0 officially support Kotlin. Our team mainly uses Spring Boot 1.5.x versions, and we come across Spring MVC v4 series. Although there are some issues, you can use Kotlin with Spring 4 series. I’d like to share a few notes on thing you need to take care in integrating Kotlin with Spring.

Making calls to Kotlin code from Java code

Here are the things you need to take care in making calls to Kotlin from Java code.

You cannot use data classes for binding request parameters

With Spring, a method annotated with @RequestMapping, and its extension, @GetMapping, corresponds to an endpoint. If you set such method’s parameters as POJO (Plain Old Java Object), you can bind the parameters to the corresponding endpoint. To use binding in Spring, the following requirements must be met. An example of this is the ModelAttributeMethodProcesse class.

  • A POJO shall have a default constructor
  • Each POJO field shall have a setter.

So if you use data classes consisting only of vals, as POJO, you will get the java.lang.NoSuchMethodException. We could meet the first requirement by the official No-arg compiler plugin, but with Spring 1.x series, we won’t be able to use nullable types like string and vals, because we need setters. We use general classes instead of data classes for DTO, var for optional data and add lateinit var to required data. Instead of using constructors, we define fields inside classes. Although you can use lateinit var in data classes, you cannot in constructors. Which means, you can use lateinit var only inside a class, but those fields will be excluded from getting automatically generated methods such as toString() and hashCode().

This issue has been resolved in Spring 5.0 series. To get a better understanding, compare the ModelAttributeMehotdProcesser class of Spring v4.x and v5.x. If you have to use Spring Boot v1.5.x and lower versions, you could try implementing an ArgumentResolver.

You cannot use @Component annotation alone

Spring automatically registers classes annotated with @Component or @Service as beans. And during the process, dynamic proxy is used to extend the class. But, by default, all Kotlin classes are final — thus cannot be extended — so unless you a modify the class with the open modifier, the class is excluded from a Spring’s component scan. For classes that need to be, or perhaps that might need to be extended, there is a Gradle plugin, kotlin-spring, that makes the classes open. If you use Spring Initializr, the dependency on the plugin is specified in the build.gradle file by default, but if you are developing from scratch, remember to take care of the component scan problem. 

Try-catch is unavailable if calls are made from Java – there are no checked exceptions in Kotlin

There are no checked exceptions in Kotlin. So when you throw an exception, even if Kotlin code throws Java’s checked exception, you don’t have to use trycatch statements. But do note that if you using both Java and Kotlin and Java code is calling Kotlin code, then your trycatch statement will trigger a compile error. To avoid this error, you can use the @Throws annotation, as shown below.

@Throws(IOException::class)fun throwSomeException(): String

If you are using Spring AOP, take care in creating a proxy for an object with CGLIB. In CGLIB, if a checked exception is thrown that is not specified with the throws, the UndeclaredThrowableException, an unchecked exception is thrown, which Spring is not capable of handling.

Calling Java methods from Kotlin – Parameters and return values

This time, let’s look into the opposite case; making calls to Java methods from Kotlin. Every data type in Kotlin is nullable, but there is no such thing in Java. From Kotlin’s point, all Java method parameters become a platform type (T!) and null-safety, one of the Kotlin’s merits, cannot be applied. To overcome this, you need to hint Kotlin by annotating Java parameters with @NotNull or @Nullable. Of course, we could cast T! to T or T? and make the caller handle it, but technically, this does not guarantee null-safety. 

Now, let’s look at return values. A possible issue is using helper methods like Mockito.any() for method mocking with Mockito. The any() method returns a null value, which would cause a runtime error if we attempt to use it in Kotline code, on a parameter set as non-null. This led us to make a little helper class as the following.

@Suppress("UNCHECKED_CAST")
class KotlinMockitoHelper {
    companion object {
        fun <T> any(): T {
            return Mockito.any() ?: null as T
        }
    }
}

There is a discussion about this; methods that return a generic type cannot do a null-check, so we need to do a null-check with a non-null type, as shown below. When you compile the following code, you might get a message from the compiler that the null check is not required. But, if you run the b() function without the null check, you will get the NullPointerException.

fun <T> a(): T {
    return null as T
}

fun b() {
    val c: String = a()
    if (c != null) { // Your compiler will display some warnings here!
        print(c.length)
    }
}

Having no need for a null-check for the methods that return a generic type somehow turned out as a workaround for mocking. There is a library for customizing Mockito for Kotlin, mockito-kotlin, including this workaround for the any() method. We are still using our helper method, having no issue so far, but if maintenance becomes a burden, we may opt for mockito-kotlin.

Entering Clova Skill Awards with Kotlin

We’d like to share our story on developing a “Clova Skill” with the following tech stack.

  • Language:Kotlin
  • Web framework: Spring Boot 2.x, WebFlux
  • O/R mapper: Exposed
  • DB: MySQL

In-house Clova Skill Awards

Developing Clova Skills (Extensions) has become available to third-party developers as well. You can write a program to speak to Clova in the language of your choice or to get the end-user’s response. Figuring out the response is the role of the Clova Extensions Kit (CEK), so writing apps for AI speakers is open to everyone.

Before releasing this platform to third-parties, there was an in-house contest to make fun Clova skills, testing the platform at the same time. A total reward of 500,000 JPY was at stake! Kagaya, who has written the first part of this post, and I joined LINE around the same time. Our job at LINE is not in anyway close to Clova, but Ryokachii, a friend of us asked us to form a team.

We developed a skill to help cooking. I am sure many of you have had an experience of cooking while checking the recipe your smartphone. You can watch the video to get the idea — sorry, it’s available only in Japanese. 

Once you add our chatbot as a friend, and register a recipe, Clova reads out the recipe for you. It comes with a timer and reads out the recipe again as your request.

So, I’d like to show you how to make a Clova skill with Kotlin. The official Clova CEK SDK for Kotlin has been released, but we went out without using the SDK. The content I’ll share here is applicable with other APIs and in any other web applications. Also, even if you have no experience in server-side, please do have a go; I am not a backend engineer myself, I’m an Android engineer.

The whole process is divided into the following four steps:

  1. Creating a project
  2. Defining model classes
  3. Handling requests for various Intents
  4. Using Exposed

Creating a project

Visit Spring Initializr, generate a Gradle Project with Kotlin and Spring Boot 2.0.4. In the dependencies field, add Reactive Web and MySQL. Next, install the Kotlin plugin on IntelliJ.

Defining model classes

Before we get into this, I’d like to ask for your understanding. Unfortunately, the CEK documentation is only available in Japanese and Korean. Based on the guide, write a request to the CEK server, and model classes for each JSON object units that consists a response. This is when the Kotlin’s data class and null-safety come in handy. With data classes, you can implement a model class easily. And, since the fields the document states as optional may or may not be sent from the Clova server to ours, we can specify an optional field as nullable by defining them as val xxx: String?. Kotlin supports annotations like Java does, so we used Jackson annotations. Here is a part of our code.

// A request from CEK
data class ClovaRequest(
    val version: String,
    val session: Session,
    val context: Context,
    @JsonTypeInfo(
        use = JsonTypeInfo.Id.NAME,
        include = JsonTypeInfo.As.PROPERTY,
        property = "type",
        visible = true
    )
    @JsonSubTypes(
        JsonSubTypes.Type(value = LaunchRequest::class, name = "LaunchRequest"),
        JsonSubTypes.Type(value = IntentRequest::class, name = "IntentRequest"),
        JsonSubTypes.Type(value = SessionEndedRequest::class, name = "SessionEndedRequest")
    )
    val request: RequestInterface
)

// One of the three
// Execution request, Intent request, or Session termination request
interface RequestInterface {
    val type: String
}

// Execution request
data class LaunchRequest(override val type: String) : RequestInterface

// Intent request
data class IntentRequest(override val type: String, val intent: Intent) : RequestInterface {
    data class Intent(val name: String, val slots: Map<String, Slot>?)
    data class Slot(val name: String, val value: String)
}

// Session termination request
data class SessionEndedRequest(override val type: String) : RequestInterface

data class Session(val new: Boolean,
                   val sessionAttributes: Map<String, String>?,
                   val sessionId: String,
                   val user: User)

data class User(val userId: String, val accessToken: String?)

data class Context(@JsonProperty("System") val system: System)
data class System(val application: Application?, val user: User, val device: Device)
data class Application(val applicationId: String)
data class Device(val deviceId: String, val display: Display?)
data class Display(val size: String?, val orientation: String?, val dpi: Int, val contentLayer: ContentLayer?)
data class ContentLayer(val width: Int, val height: Int)

Handling requests for various Intents

CEK analyzes the user’s vocal request by comparing it to the examples registered, and then send an Intent to our server. For our project, we defined the following two Intents.

  • RecipeSelectionIntent for “Please start the nth step.”
  • TimerIntent for “Let me know when m-minutes has passed.”

Now, using Kotlin’s sealed class is an easy way to handle Intents. The following is from our code.

// Intent definition
sealed class OryoriInternalIntent {


  // "Please read the nth recipe", "I'd like to make number 5"
  data class RecipeSelectionIntent(val recipeId: String): OryoriInternalIntent()
 
  // "I am ready to cook"
  object PreparedIntent : OryoriInternalIntent() 


  // "Let me know after five minutes", "Timer, five minutes"
  data class TimerIntent(val min: String) : OryoriInternalIntent() 


  // "Tell me again", "Read me the recipe again"
  object RepeatIntent : OryoriInternalIntent()


  // "Help", "Tell me how to make it"
  object HowToIntent : OryoriInternalIntent()


  // Others
  object UnknownIntent: OryoriInternalIntent() 
}

// Convert Clova Intent to the Intent defined above
@Component
class OryoriIntentMapper {
    fun fromIntent(intent: IntentRequest.Intent): OryoriInternalIntent {
        return when (intent.name) {
            "RecipeSelectionIntent" ->
                intent.toInternalIntentWithFirstSlot { OryoriInternalIntent.RecipeSelectionIntent(it) }
            "PreparedIntent" -> OryoriInternalIntent.PreparedIntent 
            ...
        }
    }
}

Now, let’s write the code to respond to an HTTP request. Firstly, define a router. Annotate the class with @Configuration, and the method that returns a router with @Bean. The Spring DI framework automatically scans them through component scan and uses the class and the method’s return value. 

With the following example, when an HTTP request sent by the Clova server via a POST method is received, the clovaHandler::handle method is called. You can see that the clovaHandler is passed as a constructor parameter. Since the class is annotated with @Component, the instance of the class is injected to the Router’s constructor argument automatically.

@Configuration
class Router(private val clovaHandler: ClovaHandler) {

    @Bean
    fun apiRouter() = router {
        accept(MediaType.APPLICATION_JSON_UTF8).nest {
            POST("/", clovaHandler::handle)
        }
    }
}

This time, we’ll write the class ClovaHandler which actually receives requests from the Clova server. By using Spring Webflux, you can write the process, “Request → Intent analysis → Response”, in a declarative manner. Webflux is based on Reactor, a type of an implementation of Reactive Streams. If you’ve ever used RxJava, you will be able to understand the following code easily.

@Component
class ClovaHandler (...) {
    fun handle(request: ServerRequest): Mono<ServerResponse> {
      // Deserialize ClovaRequest class and receive
            return request.bodyToMono(ClovaRequest::class.java) 
                .doOnNext {
                    logger.info("{}", it) // For debugging
                }
                .map {
                    // it represents a ClovaRequest
                    // You can use "when" and "is"
                    when (it.request) {
            // Request to execute the skill
                        is LaunchRequest -> launchRequestHandler.handle(it.session.user.userId) 
                        is SessionEndedRequest -> SpeechesSession(SpeechInfo("Call me again when you cook next time."))
                        is IntentRequest -> {
                            val oryoriIntent = oryoriIntentMapper.fromIntent(it.request.intent)
                            assistant.assist(oryoriIntent, it.session.user.userId)
                        }
                        else -> throw RuntimeException("Unknown request type is given")
                    }
                }
                .map {
                    // Make Clova deliver a message — a response for the given Intent — to the user
                    val outputSpeech = OutputSpeech(SpeechType.SPEECH_LIST, it.speeches)
                    ClovaResponse(shouldEndSession = it.endsSession, outputSpeech = outputSpeech) 
                }
                .flatMap {
                    // Serialize ClovaResponse object to JSON
                    ServerResponse.ok() 
                        .contentType(MediaType.APPLICATION_JSON_UTF8)
                        .body(Mono.just(it), ClovaResponse::class.java)
                }
    }
}

Using Exposed

For saving user information and recipes, we used MySQL, but for the O/R mapper, we tried Exposed, 100% written in Kotlin. Exposed is a light SQL library made by JetBrains. To use it, add the following dependency in Gradle.

compile 'org.jetbrains.exposed:exposed:0.10.2' // The version we used for our project
compile 'org.jetbrains.exposed:spring-transaction:0.10.2'

This is how you would define a table schema for Exposed, with Kotlin.

object Users : Table() {
    val id = varchar("id", 255).primaryKey()
    val recipeId = (integer("recipe_id") references Recipes.id).nullable()
    val step = integer("step")
    val createdAt = datetime("created_at")
    val updatedAt = datetime("updated_at")
    val status = varchar("status", 512)
}

create(Users)

The code for searching a value looks something like this. The following code searches for a user, with an ID. 

fun findUser(userId: String): User? = Users.select { Users.id eq userId }
    .limit(1)
    .map { it.toUser() } // SQL result is ResultROW, so convert it to the User class 
    .firstOrNull()
}
private fun ResultRow.toUser() = Users.rowToUser(this)

// DAO definition
data class User(val id: String, val recipeId: Int?, val step: Int, val status: String)

To manage transactions with Spring’s @Transactional annotation, we define the configuration as the following. We assign SpringTransactionManager of Expose, to the PlatformTransactionManager which we use for managing transactions with @Transactional. In this way, while using Exposed methods, we can start, commit, and rollback transactions in units of methods annotated with @Transactional

import org.jetbrains.exposed.spring.SpringTransactionManager
import ...
@Configuration
@EnableTransactionManagement
class TransactionConfiguration(val dataSource: DataSource): TransactionManagementConfigurer {
    @Bean
    override fun annotationDrivenTransactionManager(): PlatformTransactionManager =
        SpringTransactionManager(dataSource)
}

My experience with Exposed was satisfactory. Defining and writing tables were easy, being able to write SQLs like CRUD and JOIN with Kotlin was quite useful. Support for type inference is another strength of Exposed. One downside was that I had to know how to write the SQL Exposed uses; it may take a while for me get used to it.

Many projects at LINE use MyBatis which supports SQL. You can use Kotlin with MyBatis too. If you add the all-open plugin and no-arg compiler plugin, you can use data classes for DAO (Data Access Object), like shown below. Although it is convenient, such data classes behave differently from original data classes.

data class User(val id: Int? = null, name: String)
val user = User(name = "foo")
insert(user)
user.id != null  // if true, an ID generated in DB is assigned

Lastly

Although it was very swift, we shared how we developed a Clova skill with Kotlin and Spring Boot 2. I have to say, Kotlin is fun! Why don’t you have a go yourself?  Start trying applying Kotlin in your project, from small parts.