Q. Hello, Trustin! Let's start with a brief introduction of yourself.
A. Hi. My name is Trustin, and I'm currently involved in open source development of Armeria and Central Dogma at LINE. Before I joined LINE, I worked at Twitter, responsible for the maintenance of Netty as well as the development of new features and upgrades to next versions.
Q. You seem to have built your reputation as an open source developer with the Netty project. Do you have any updates on the Netty project that you want to share?
A. I handed over the lead role of the Netty project to Norman, who currently works at Apple. We have worked together on the Netty project for a long time, and he is managing the project very well even as we speak. Yet, since there is a lot of my work remaining with Netty, I still play a part in the project by reviewing pull requests whenever I find time.
Q. I see. Do you happen to have a long list of PRs to review as of now?
A. PRs are always there no matter what. :) Since Netty is a low-level project, more senior developers tend to participate. So they are aware that PR reviews can take a longer time with this type of project, and that some PRs can be rejected if they are not in line with the project. Unlike the old days, more developers are now well acquainted with the development process of open source projects.
Q. You have been working as an "employed" open source developer at Twitter as well as at LINE. Have you ever experienced a dilemma between what you think was right as an open source project lead and how the company expected the project to evolve?
A. Recently companies approach open source projects from a genuine engineering perspective. So, they grant a high level of discretion to "hired" open source developers in terms of how to implement technological changes. Probably this is why I have not actually faced a critical dilemma.
Q. Why do you think companies choose to hire open source developers?
A. As anyone can contribute to open source projects, companies can implement what they want by submitting a pull request without going through the inconvenience of hiring open source developers. In fact, before I joined Twitter, another engineer working at Twitter had contributed to implementing the SPDY protocol in Netty. Technically speaking, there is no need for companies to hire a programmer to implement new features in an open source project. In reality, however, it is not easy to find a programmer who is an expert in the respective areas. So, if a company plans to hire someone, it is only natural for the company to hire someone who is already participating in the project. When I was working at Twitter, I implemented an asynchronous DNS (Domain Name System) and SOCKS protocol upon request from in-house engineers or on an on-demand basis. One of the biggest advantages of having an in-house open source programmer is that you can directly ask for what you need to be implemented.
Q. In the past, open source projects used a traditional form of mailing lists as a communications channel. How is it nowadays?
A. It varies case by case. Those projects with a long history still use a mailing list while younger projects no longer tend to use a mailing list. Recently Slack or Gitter are more common. For instance, Netty guides programmers to post their questions on Stack Overflow while Armeria ask users to register their questions as an issue on GitHub. When questions are submitted as a new issue for the Armeria project, one of the project members including myself labels it as a question and groups them together to allow easy access to other contributors. When the Armeria project evolves further in scale, we might have to think of other ways. Up to now, I think this approach works fine as long as it does not undermine the inherent purpose of GitHub as an issue tracker.
Q. Can you tell us some of the difficulties you experienced with open source projects?
A. Once in a while, you come across a project that achieves success in a short period of time. However, most of them take a long time to become a successful one. Let's take Netty as an example. It took quite a long time to put Netty on track. Despite benefiting from Apache MINA's fame as its sequel, it took a number of years before enough contributors participated in the Netty project. That's why patience and perseverance are important virtues when it comes to open source projects.
Q. Let's say your work can be broken down into coding, code review, and others. How much time do you allocate to each category?
A. I spend most of my time on code review, followed by coding.
Q. What would be the most frequent comments you make for code reviews?
A. I start from an overall structure and dive deep into the code for optimization from a readability perspective. For code readability, I make comments to make the code look prettier (more readable). I also make suggestions on naming or indentation conventions. Another area for comment is error message coding conventions. There are implicit rules for the style of error messages for specific libraries. Users could get confused if an error message is displayed in a different format.
Q. How do you handle disagreements during code review?
A. First, I agree to disagree, admitting that we have different opinions. Each opinion has pros and cons so I try to narrow the gap as much as possible to finally make a choice from a technological perspective. To be frank, I think to myself, "I still think I'm right but if you insist..." :)
Q. When it comes to communications in writing, I sometimes feel it would be easier if I could talk face to face. What do you think about that?
A. I personally prefer putting down my thoughts in writing to make my points clear and logical. When I talk, sometimes words don't come out the way I want or they are just not clear enough. And, not all of open source project contributors are comfortable enough with conducting real-time communications in English. Moreover, others who are not part of such conversations cannot know what is being discussed. In the end, even if we meet offline to make certain decisions, they have to be written down for record-keeping purposes. I believe communities can be well maintained when there is a guarantee on transparent communications.
Q. Now, before we move on to questions about Armeria, some say a developers' hobby is development. Does this also apply to you?
A. HAHA. I could say that for me in the past. Nowadays I spend my free time with my family. Sometimes I watch developer conferences on YouTube while exercising on a Xiaomi WalkingPad.
Asynchronous microservice framework, Armeria
Q. Did you start the Armeria project after you joined LINE? Can you tell us how the Armeria project came to life?
A. I laid the groundwork for low-level frameworks with Netty, and I only had a vague idea about developing something high level on Netty to make server development easier. However, I didn't have concrete ideas about what because Netty kept me pretty busy back then. :) When I came to LINE, I saw specific needs for such frameworks, and that's when I fully took on the initiative.
Q. Can you briefly explain what Armeria is?
A. Netty is a general purpose low-level framework, supporting various protocols. On the other hand, Armeria is a web-based microservice framework built on top of Netty. Armeria allows engineers to build a high-performance asynchronous reactive web service more easily. Armeria supports HTTP/1 and HTTP/2 as a session layer protocol as well as the HAProxy protocol. In terms of the application layer, it is the only open source framework that supports both gRPC and Thrift in addition to REST API. Moreover, you can run existing Java EE (Enterprise Edition) web applications at the same time, allowing a gradual migration from legacy systems. Armeria also offers various features that are essential for building microservices inlcuding service discovery, circuit breakers, client-side load-balancing, and metrics.
Q. Synchronous APIs are still widely used when implementing web services, but Armeria is more focused on the asynchronous architecture. Why?
A. When a web service using synchronous APIs on the traditional web framework grows in size, it will inevitably face limitations if it solely relies on synchronous frameworks. This is why it is natural to evolve toward asynchronous or reactive paradigms along with the growth of services. Such growth in scale calls for a web framework that supports reactive programming or asynchronous paradigms, just like Armeria.
Q. Can an asynchronous framework process more requests?
A. The answer to this question may vary case by case. Generally speaking, asynchronous frameworks tend to become more beneficial the longer the storage response time. In particular, when a specific back-end problem or overload slows down the response time, this could scale up to overall system failure as other requests that do not go through that back-end are still affected under a synchronous structure. On the other hand with an asynchronous framework, problems caused by a specific back-end can be contained without special tuning, limiting the impact on overall system performance. This makes a big difference especially when distributed data storage is a mainstream trend.
Q. Still, more are using synchronous APIs. Why do you think asynchronous APIs haven't been able to take over yet?
A. First of all, it requires a certain size to benefit from an asynchronous architecture. In other words, it would not make much difference between synchronous or asynchronous structures if there are only about 100 users accessing the service. When server performance and memory size are sufficient, services with synchronous APIs would process requests without a problem. It is when much more users begin to access the service. Particularly, services with synchronous APIs could face limitations when a large number of users try to access the service within a specific time frame. This issue can be easily resolved with the asynchronous architecture. Considering these potential issues, when a service is built with an asynchronous framework from the beginning, it will be able to handle growing requests smoothly. That's probably why more programming languages or libraries support asynchronous frameworks recently. Node.js, widely used nowadays, can be a good example. I think more and more developers have a growing interest and deeper understanding of asynchronous processing or reactive programming.
Q. Can't we create asynchronous API on the Spring framework?
A. It was difficult in the past. However, with the emergence of Spring WebFlux, we can now build asynchronous or reactive web applications on the Spring framework. Come to think of it, the Spring framework can be a good indicator of growing interest in asynchronous processing. In the past when an asynchronous architecture was still an unfamiliar concept for developers, development on the Spring framework was more or less limited to synchronous processing. The fact that the Spring framework now supports asynchronous processing testifies to the recent trend among developers, increasingly adopting an asynchronous approach with the growth of web services in scale.
Q. Many LINE services use Armeria. Have you given some thoughts as to Ameria's road map vs. requirements from LINE services in terms of priority or future direction?
A. Open source projects are mainly driven by feedback from their users, who are usually developers that use the respective software. So, whatever the feedback, user feedback is essential to the project. Like the old saying, "The opposite of love is not hate, but indifference." I think it is important to quickly respond to user requests, implement required features and keep a good balance with the overall road map. There is no specific deadline for tasks with open source projects. There can be a "dream" road map in mind, but certain development can take a priority to meet user demand. Yet, it is important to stay within the boundaries of basic principles of the project despite user demand. Luckily, we don't usually receive such far-fetched requests from users because they usually understand the scope of the project.
Q. Competition often fuel the growth of services like Uber and Lyft. Which service would be a main competitor for Armeria? Can it be Spring WebFlux mentioned earlier?
A. It can be seen as a competitor but, at the same time, as a partner in some sense. Asynchronous programming is expanding its presence but still lags behind synchronous programming. In this sense, it would be mutually beneficial for players in the asynchronous ecosystem to offer more choices. In addition, Armeria focuses on helping users to build web services, in particular microservices, while Spring provides a wider array of functions. That's why Armeria offers a module to make an integration with Spring easier. For instance, you can use Spring WebFlux, a reactive version of Spring Web, and adopt Armeria for the HTTP layer. There are many applications built on Spring at LINE. Usually, Java EE web applications are built on Spring with Armeria supporting Thrift or RPC.
Q. Why is Armeria called Armeria?
A. The Armeria project was initially kicked off to address a scalability issue at LINE by changing the existing Thrift RPC stack to a more modern HTTP/2 based asynchronous framework. The keyword here was "Thrift". When you look up thrift in the dictionary, it has a meaning of frugality but it is also another name for a wild flower called Armeria. That's how we picked the name, Armeria. The Armeria project logo image represents three flowers, each flower representing a microservice. The meaning behind this logo is that many microservices come together to offer a big scale service.
Come and work with me at LINE!
Q. What made you decide to join LINE?
A. After working at Twitter for a few years, I wanted to go for a new challenge, not that I didn't enjoy the maintenance of Netty. I started to look for new opportunities and I found out LINE had a lot of development teams and many interesting projects underway. I thought I could settle in smoothly, especially since some talented acquaintances of mine were already working at LINE. :)
Q. What is the profile of people you want to work with?
A. My team covers both Armeria and Central Dogma. Open source development experience is preferred but not required. Those who have experience in framework or library development would be great. Also, a deep understanding of the Java language is a plus as it is required when designing new APIs. I also like a detail-oriented person. For instance, a single PR can entail a series of changes here and there. I like a programmer who will implement relevant changes throughout the code without specific instructions rather than sticking to specific comments from the PR. Lastly, I want to work with someone who is fluent in "text" communications. When an instant messenger is a main channel of communications, it often becomes "asynchronous" so I expect my counterparties to express their thoughts logically in writing with the details of cause and effect.
Q. It is not easy to know if someone is detail-oriented or not until you actually work with that person. How do you verify this?
A. You're right. It is difficult to verify that only with a basic level of coding tests. That's why I also review their GitHub portfolio or a history of PRs sent to other projects. If that person is leading his or her own open source project, it makes verification all the more effective.
Q. Any last words before we wrap up this interview?
A. Many people ask me for advice, saying that they want to contribute to open source projects but they are not sure which project they want to contribute to or how. What's important is to stop asking and start with what you know and push yourself from then on. You need to start doing something to see where you want to go. It won't help much just wondering about which is the best path for you. Why don't you start with creating PRs for Armeria or Central Dogma right now? :)