LINE with OpenJDK: From compatibility check to things to watch out for

Good to see you all! I’m YongChan Kwon, an SRE (Service Reliability Engineer) at the Service Engineering Team. As you can guess from the team name or my role, I’m more like a generalist, filling in the gaps between various stages in service life cycle such as infrastructure, development and service technology.

As Oracle announced its new Java license model in 2018, effective January 2019, Oracle JDK is no longer free. Against this backdrop, LINE has set up an internal task force (TF) to review and define requirements for potential migration to OpenJDK. I’m writing this post to give you an overall summary of TF’s progress and technical aspects considered.

 

Getting started

Let me start with an old tale, as my engagement in the OpenJDK review brought back some old memories. In the early 2000s, opinions about the reliability of open source software (OSS) were extremely divided, and it was not realistic to adopt open source software, especially at public institutions or large companies. For instance, back then, Apache HTTPD Server’s prefork MPM (Multi-Processing Modules) was known for its memory leak problem and newer version called worker MPM was yet to establish a reputation for stability. Consequently, most of public and commercial web sites adopted Iplanet web server, and I humbly confess that I’m an author to one of many books on Iplanet web server topic (Time sure flies!). Nowadays it might sound “insane” to pay for web servers. However, in the old days, we didn’t have many choices (Web servers priced at around USD 1,500 per CPU). During these times, Iplanet web server had no competition for general web service as it was a multi-process, multi-threaded web server and came embedded with Java servlet container. With evolving web service environment, the textbook approach, back then, was to implement a 3-tier architecture with Iplanet web server and WebLogic.

In the next decade, Apache worker MPM (multi-process + multi-thread) dominated the web server world before various lightweight web servers including Nginx with the event-driven architecture took over the scene. After the fall of EJB (Enterprise JavaBeans), Tomcat servlet container became a mainstream trend to provide stable service. Thinking back to the old days, companies using a commercial infrastructure software would ask the following questions about OSS products:

“Who’s going to take care of bugs, configurations or failures?”
“What if it simply stops its service? How do we upgrade?”
“Would it deliver expected performance?”

I used to advise those who were considering to adopt OSS:

“Your organization is ready to adopt OSS when people in your organization change their perception about OSS.”
“Stop asking who’s going to take responsibility. Take the initiative to fix the bugs yourself instead of relying on external support.”

In reality, there were limitations at public organizations or large companies because many had a job rotation program in place and people were tied up with managerial workload after promotion. Dot-coms or, more recently, startups became more ready to adopt OSS. Moreover, installation or configuration of web servers or servlet containers no longer require state-of-art technology. Still, I feel like people are asking the same questions again after 20 years when they start to turn their eyes to OpenJDK, following Oracle’s announcement on the new license policy.

  • “OpenJDK? Who’s going to take care of it when there is a problem?”  → As always, there can be a bug or a problem. But, at the same time, every problem has a solution: Try and test a different version, and you’ll soon find a solution.
  • “Is it going to be upgraded?” → I have no concern about upgrades. I think OpenJDK will still be there even if Oracle is not.
  • “Will it provide comparable reliability and performance?” → There are many distributions of OpenJDK released after completing the Technology Compatibility Kit (TCK) led by Oracle, which ensures Java compatibility. For performance or reliability issue, we plan to conduct an internal test before adoption.

As I was browsing through official/unofficial documents and online postings on the Internet, I found it interesting that people still had these concerns when I rarely paid Oracle including formerly Sun Microsystems for technical service throughout my career of developing and supporting Java infrastructure products since Java v1.3. I’m not an OpenJDK evangelist, but open source environment has earned trust in the market and there are many services built on open source products when you look around. Introducing OpenJDK can be perceived as adapting to an open source environment. Yet, there are still a few things to review. 

 

What to consider when you plan to adopt OpenJDK

To make a long story short, it didn’t give us a big headache. It sure took a long time to test out different versions of OpenJDK, but we didn’t face any serious problem or performance discrepancy. I’ll elaborate later, but our focus was more on “productivity” and “costs”. 

The main goal of the OpenJDK TF was to assess migration from Oracle JDK to OpenJDK, especially since Oracle would not provide upgrades from January 2019 even if bugs or security vulnerabilities were found in Java 8 Oracle JDK. We didn’t consider upgrading major Java versions. Preparing for migration to OpenJDK is a whole different level, compared to upgrading from Java 8 to Java 11. For a major version grade, it involves handling deprecations in the application code, framework compatibility, new code style and additional development on top of replacing the software (JDK). So, it is more of application code issue than compatibility and performance of JDK. 

For your information, we didn’t worry much about versions scheduled for EoS (End of Support) or other support issues as the OSs used in LINE systems are well kept up to date. If you have an old version server or a recent one that you cannot update, I personally recommend to “maintain the current setup and NOT update JDK”*. I think it is better to adopt OpenJDK when you replace your old server with a new one.

When you are moving from existing Oracle JDK to OpenJDK, you need to consider which distribution version to choose and how compatibility, stability and performance can be tested. You can study the relevant documentation or use a specific tool to conduct Java load testing. 

There are many distributions of OpenJDK. Once a JSR (Java Specification Request) becomes final through the JCP (Java Community Process), different distributions of JDK are provided in line with the JSR reference implementation and tested with the TCK (Technology Compatibility Kit) for compliance. Long time ago, Sun Microsystems provided and disclosed the JDK code to the OpenJDK community, except for a few modules (example: JFX) with license issues. Many developers distribute OpenJDK, using or improving the disclosed JDK code. For example, as a next-generation garbage collector (GC) of the Garbage-First (G1) garbage collector, Oracle offers the ZGC and Red Hat provides the Shenandoah GC. Both ZGC and Shenandoah GC are provided for OpenJDK. In particular, when you install Red Hat’s OpenJDK, backports to JDK 8 and JDK 11 are available as well. As both GCs are not yet in the final stage, there are many possible scenarios going forward, depending on how much progress each makes, such as the OpenJDK Community using both GCs or different providers making their own choice.

It is somewhat tricky to completely segregate stability and performance for testing. As a compromise, we set the target and conducted tests to see if the target can be reached. In short, we ran the program for a long time (more than 10 hours) for stability test while we installed OpenJDK on some of the servers in operation to actually handle user requests to reach the target level (that is, three times the average).

List of OpenJDK distributions

The following table shows a list of OpenJDK distributions available with a brief description of their characteristics.

Community/VendorProduct NameOSS/CommercialArchitectureDescription
ORACLEOracle JDKCommercialHotSpotRequires a subscription to use Java 8 from January 2019.
Note. This distribution was used as a basis for comparison
AzulZingCommercialZingJVM with Azul’s proprietary architecture. This was originally a commercial distribution.
Note. This distribution was excluded from comparison because it is a commercial distribution with a different architecture
OpenJDK communityOpenJDKOSSHotSpotOpen JDK community is a group created since the era of Sun Microsystems, a birthplace of OpenJDK as we know. It mainly serve as a community for OpenJDK.
Source code distribution is available but OpenJDK binaries can be found at https://jdk.java.net. Binaries are provided at https://jdk.java.net are for reference purposes only (https://jdk.java.net/java-se-ri/8).
AzulZuluOSSHotSpotOpen source JDK provided by Azul with a different architecture from Azul’s Zing. Provides a certified build of OpenJDK. Paid support options are available from Azul.
Red Hat/CentOSOpenJDKOSSHotSpotDistribution by Red Hat, available as RPM (binary) from RHEL (Red Hat Enterprise Linux) or CentOS yum repositories. Supported versions vary a little, depending on an OS version. (cf. Windows binary)
AdoptOpenJDK.netOpenJDKOSSHotSpotOpenJ9(IBM)This community of OpenJDK advocates run with donations from many corporate sponsors. This one provides binaries of HotSpot (OpenJDK) led by the OpenJDK community with support from Oracle and of OpenJ9 VM led by Eclipse with support from IBM available. AdoptOpenJDK provides the most diverse platforms and binaries of VM architecture (including Mac OS for PCs).
Eclipse.orgOpenJ9OSSOpenJ9(IBM)Launched with IBM’s donation of its own JDK, using OpenJ9 as a JVM. OpenJ9 can be an option for companies that are considering to shift from IBM JDKs to OpenJDK. There is discrepancy in configurations and architecture compared to HotSpot.
Note. This distribution was excluded from comparison for such discrepancy issue

 

Checking compatibility

Code compatibility

The question of code compatibility between Oracle JDK and OpenJDK might sound groundless, but there can be an issue. Generally, ordinary web-based applications shouldn’t have a compatibility issue if they passed the TCK test (putting performance or other issues aside). It was also the case for the OpenJDK TF. Yet, we found out that JavaFX data type objects were used in the code. (JavaFX is not included in OpenJDK due to a license issue. OpenJFK can be added instead.) When we started our WAS, a number of exceptions appeared in the logs, which was caused by using javafx.util.Pair and resolved by changing the code to use a different type of object.

Caused by java.lang.ClassNotFoundException: javafx.util.Pair  

For JavaFX, to resolve this issue, separately build OpenJFX and add to the OpenJDK classpath or simply delete the relevant code.

JVM (Java Virtual Machine) compatibility

Some JVM options might not be available, depending on the architecture of a JVM. For instance, if you were using -XX:+UseG1GC to use the G1 GC in the Oracle JDK environment, your WAS might not start with Eclipse OpenJ9. This is why it requires more consideration before moving to Eclipse OpenJ9 or Azul Zing as they use a different architecture. This is also why the OpenJDK TF excluded Azul Zing and Eclipse openJ9 for the following compatibility test.

  1. Check if the WAS with OpenJDK would run with the existing JVM options:
    • Confirm if the VM is properly initialized.
    • See if there is any incompatible option.
  2. Deploy and start the WAR on the WAS with OpenJDK and check if any exception occurs:
    • Assess if any of the classes used previously is missing (Example: Java Cryptography Extension)
    • Check if any of the classes unique to Oracle JDK (Example: JavaFX) is used
  3. Conduct monitoring after installing Alpha or Beta WAS: Look for any unconfirmed issues

 

Checking stability and performance

I experienced an unexpected JVM failure in the past after a Java minor version upgrade. It turned out that JVM’s internal bug caused a process crash. I rolled back to the previous version and waited for the next update to avoid this problem. We always hope our hardware and software are defect-free and bug-free. However, the reality is that any hardware can break down and any software can have a bug. That’s why we turn to commercial products to get the necessary support when there is a problem and we also sign a maintenance contract as an insurance. 

As for open source compared to commercial packages, information and patches become available faster. Of course, there is an important prerequisite; you as a company need to have internal resources to handle the technology. Commercial product providers offer a critical patch update (that is, hotfix) from time to time, but they usually wait for next regular patch or version upgrade instead. Companies without in-house resources could find such regular updates more stable. Fixes can be provided for problems found soon after an update/upgrade, but some problems only emerge after some time when load piles up with an increase in the number of threads or memory usage. Since such problems can be identified some time after the installation or upgrade, it can have a critical impact on the service. Rolling back the JDK on hundreds of servers is not only a matter of service quality but also a matter of engineers’ workload. So we conducted the following tests on OpenJDK, following compatibility check.

Using JMH (Java Microbenchmark Harness), we ran the benchmarking program for more than 10 hours and conducted the following tasks using code samples. It takes about 10 to 14 hours for all of 38 code samples to be executed.

  • Check if the JVM operates normally, using a benchmarking program and samples provided by the OpenJDK community
  • Run the benchmarking program for more than 10 hours to check the JVM; checking for occurrence of process down, GC logs and etc.
  • Use the benchmarking data to compare between different OpenJDK distributions

The results showed only a small discrepancy in performance between different distributions. Conditions (load) for code samples given by JMH were not modified. Total of 38 code samples and string operations were conducted, of which some results are shown below. The following test results are for illustrative purposes only to compare the performance between different OpenJDK distributions. These test results are subject to change under a different test environment.

Benchmark test – multi-thread

This test was conducted to compare performance based on the thread scope (shared/unshared). The results are shown in terms of operations per second (ops/s) and a higher number indicates better performance. The degree of discrepancy is similar between thread scope. All three distributions showed similar results in the total number of threads, except for Adopt OpenJDK.

Benchmark test – loop

This test was conducted to compare performance, using various loop. Lower number indicates better performance.
The graph for 100 loops shows a bit of discrepancy, but it is in nanoseconds and the result for 100,000 loops appears to be similar between distributions.

Benchmark test – blocking queue interrupts

This test was conducted to compare process time by interrupting BlockingQueue. Shorter time indicates better performance. The unit is in nanoseconds, and the result showed a deviation of 1 millisecond. Rerunning the test resulted in similar result as shown below.

Benchmark test – string concatenation

This test was conducted to compare string operations, using string concatenation (+) operator and the StringBuilder class. Smaller number indicates better performance. As commonly known, the concatenation operator (+) showed significantly lower performance compared to the append() method while using append() had a better result than the concatenation operator for Zulu. But, the difference is in microseconds so it was not pointed out as a big issue.

User request handling test

Actual user requests were handled by some of the servers. We checked if the WAS with OpenJDK operates without a problem, and tested under a stress situation (three times the average load) to check performance. The following graph was built with LINE’s internal monitoring tool. The test results confirmed that the systems ran without an issue on all OpenJDK distributions. 

Partial test on live services

OpenJDK was deployed to some of the servers in service for testing. In the last stage of testing, LINE designates some of the servers as Canary Group before going fully live with all the servers. We deployed OpenJDK to those servers in the Canary Group for testing and monitoring. As of February 2019, we have not experienced any issue.

 

Provisioning

LINE uses PMC, a provisioning tool, to configure its servers by installing JDK and connecting through softlink to set the JAVA_HOME environment variable. This is a cost-saving and time efficient way to create or add new servers and set configurations. There were some limitations to this review as OpenJDK was deployed only to a part of the entire framework. The OpenJDK TF couldn’t make adjustments to the existing framework as it could undermine the current service. Still, even if it is only a part, it is not easy to manually work on tens of servers. For one server, it takes at least 15 minutes to access the terminal, copy, install, change the link, copy the required jar files and reboot. If any one of these is skipped, the entire work has to be redone. When the scope is expanded, it could involve hundreds of servers. 

Against this backdrop, we chose Ansible, which is still available as open source after the acquisition by Red Hat. I won’t go into details about Ansible but some of its strengths are as follows:

  • Playbooks expressed in YAML format for multi-machine deployments
  • Easier automation using YAML format (although actual work is run in Python)
  • Allow a dry-run (test run). It’s IMPORTANT to test before full-scope deployment.
  • Ansible module throws an exception when it fails to complete the required process.

We resolved the multi-machine deployment issue including installation and configuration through automation with Ansible. I can’t explain all the details here, but, there still remained a problem. LINE uses Kerberos authentication to grant server access, and it requires the root ID privilege to install OpenJDK under the /usr directory. We couldn’t ignore the security principles just to install OpenJDK. But, at the same time, it was necessary to grant server access and allow a non-root user to deploy OpenJDK so that it would be possible to track user activities for problem-solving purpose. I could have set it up only for me to handle everything for the OpenJDK review, but it was not realistic to handle tens of servers on my own. Moreover, we needed to set up a platform for full-scope OpenJDK deployment for the future, just in case. Given these considerations, we decided to adopt AWX, an open source version of Ansible Tower. Please check the website for more information on AWX.

I’ll share the issues identified during this process and respective solutions here. (Please understand I can’t disclose all the code and configurations.)

Issue A – Key distribution

You need to distribute an SSH public key to destination servers first, for Ansible to work. If it is only a few of servers, you can manually do the work through a terminal. However, if you are working with tens or hundreds of servers, automation becomes meaningless because Ansible won’t be able to access a destination server without an SSH public key. In other words, you are facing a dilemma where you want to use Ansible to distribute an SSH public key to destination servers but you can’t because Ansible can’t access a destination server without an SSH public key.

Now, we had to store an SSH public key at a destination server. But, it could lead to a security breach if we use ID and password to access from Ansible playbook. To avoid this issue, we used the survey feature of Ansible AWX to collect Kerberos ID and password of each user, generate a keytab and access the destination server whenever playbook is executed. By doing so, we could avoid disclosing ID and password in the code as users give their ID and password every time they access; this also serves as an audit trail.

Issue B – Automating creation of inventory

Ansible AWX lists target servers in its inventory. It is cumbersome to write a host file in the command line, and it is also a lengthy process to add multiple servers to the inventory, using the AWX GUI. Moreover, it could create a huge headache to reflect LINE’s GroupSet and Group, an in-house service classification concept for its services.

We used the AWX’s inventory script to resolve this problem. We wrote Python code to get the server data extracted from the API server and import this data into the AWX inventory.

The following is a sample of the Python code after excluding internal data for security purpose. The code might not run as it is. I included this sample to help you understand parsing grammar.

#!/usr/bin/env python
import urllib
import json
import sys
from collections import OrderedDict

# The code below is provided after excluding LINE's internal data. This might not run as it is. 
# If you can extract a server list in JSON format via API, you need to customize the code for your own use.
# You can refer to processing file-based hosts file in the section below.
# https://docs.ansible.com/ansible/latest/dev_guide/developing_inventory.html
API_SERVER_URI="YOUR_API_SERVER"
PROJECT_ID = "YOUR_PROJECT_NAME"
PROJECT_PHASE = "YOUR_RELEASE"

API_BASE_URL="https://" + API_SERVER_URI + "/" + PROJECT_ID + ":" + PROJECT_PHASE

def getJson(url) :
    url = urllib.urlopen(url)
    data = url.read()
    parsed_data = json.loads(data)
    return parsed_data

def getGroupSets():
    group_sets_url = API_BASE_URL  + "/groupsets"
    return getJson(group_sets_url)

def getGroups(groupset_name):
    groups_url = API_BASE_URL  + "/groupsets/" + groupset_name + "/groups"
    return getJson(groups_url)

def getNodes(groupset_name,group_name):
    nodes_url = API_BASE_URL  + "/groupsets/" + groupset_name + "/groups/" + group_name + "/nodes"
    return getJson(nodes_url)

# This is to print data to allow parsing by the inventory script of Ansible AWX or Ansible Tower.
# Server list is stored into Groups, which are grouped into GroupSets.
# To add a Group under a Group, 'children' entity is used.
# You won't be able to use the following code as it is. Please refer to the code for analyzing format in AWX.
inventory = OrderedDict()
inventory['all'] = {}
inventory['all']['children'] = []

parsed_groupset_data = getGroupSets()

for groupsets in parsed_groupset_data:
    groupset_name = groupsets["name"]
    inventory[groupset_name] = {}
    inventory[groupset_name]['children'] = []
    inventory['all']['children'].append(groupset_name)
    parsed_groups_data =  getGroups(groupset_name)
    for groups in parsed_groups_data:
        group_name = groups["name"]
        inventory[groupset_name]['children'].append(group_name)
        inventory[group_name] = {}
        inventory[group_name]['hosts']  = []
        parsed_nodes_data = getNodes(groupset_name,group_name)
        for node in parsed_nodes_data:
            node_name = node["name"]
            falg_maintenance = node["attributes"]["state"]
            if falg_maintenance == "NORMAL":
                inventory[group_name]['hosts'].append(node_name)

# When what's printed below reconciles with AWX parsing grammar, an inventory will be created.
print(json.dumps(inventory, ensure_ascii=False, indent=3))

 

Tools used for testing

The following is a list of tools used for testing, followed by a brief description of each tool.

JMH (Java Micro benchmark Harness)

JMH is a benchmarking tool for JVM, provided by the OpenJDK community project. It was used for stability and performance test for different JVMs. JMH can be installed as follows:

# mvn archetype:generate \
-DinteractiveMode=false \
-DarchetypeGroupId=org.openjdk.jmh \
-DarchetypeArtifactId=jmh-java-benchmark-archetype \
-DgroupId=org.samples \
-DartifactId=openjdkJMH \
-Dversion=1.0
# cd opnjdkJMH
# mvn clean install

Here is the script for selecting the JDK by setting the environment variable, $YOUR_HOME.

#!/bin/bash
  
VENDOR=$1
YOUR_HOME="YOUR JVM PATH"
    if [ "$VENDOR" = "" ]
    then
       echo "ERROR: Need Vendor NAME [oracle|redhat|adopt|azul]"
       echo "Use : setjdk.sh \$VENDOR"
       echo "ex : setjdk.sh oracle"
       exit 1
    fi
  
    case $VENDOR in
        oracle)
        export JAVA_HOME=${YOUR_HOME}/jdk/jdk1.8.0_181
        export PATH=$JAVA_HOME/bin:$PATH
        ;;
        redhat)
        export JAVA_HOME=/usr/lib/jvm/java-1.8.0
        export PATH=$JAVA_HOME/bin:$PATH
        ;;
        adopt)
        export JAVA_HOME=${YOUR_HOME}/jdk/jdk8u181-b13
        export PATH=$JAVA_HOME/bin:$PATH
        ;;
        azul)
        export JAVA_HOME=${YOUR_HOME}/jdk/zulu8.31.0.1-jdk8.0.181-linux_x64
        export PATH=$JAVA_HOME/bin:$PATH
        ;;
        *)
        echo "Need Vendor NAME [oracle|redhat|adopt|azul]";;
    esac
  
    echo "=================================="
    echo "JAVA_HOME="$JAVA_HOME
    echo "JAVA_VERSION="$JAVA_VERSION
    `$JAVA_HOME/bin/java -version`
    echo "PATH="$PATH
    echo "=================================="

And the following is a sample of script (Modifying JDK type and output data file name)

#!/bin/bash
. ./setjdk.sh oracle
$JAVA_HOME/bin/java -jar ../target/benchmarks.jar \
-rf csv -rff "EL7_All_oraclejdk_1.8_`date '+%Y-%m-%d_%H%M%S'`.csv" \
-jvm "$JAVA_HOME/bin/java" \
-jvmArgs "-Xms4g -Xmx4g -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=45" \
-prof perfnorm

JDK Mission Control

JAVA Mission Control (JMC) is provided as a commercial service by Oracle JDK. The open source version of the JMC is JDK Mission Control (JMC) led by the OpenJDK community project. As of February 2019, JMC 7 Early-Access Builds is still in development but available with basic monitoring functionality. We used the JMC for real-time monitoring of GC data and thread status of the JVM.

GC Viewer

GC Viewer is a log analysis tool for the GC data of the JVM. This is also an open source GC log analysis tool available for the G1 GC. The UI might not be as good as commercial packages but it is sufficient for data analysis. The following image is a sample of memory leakage analysis from another project as we didn’t have any issue for analysis with this review. The Git repository of GC Viewer is here, and you can find its binary here.

Ansible AWX

Ansible AWX is an open source version of Ansible Tower, which helps to manage Ansible provided in the CLI format. It was released for the use in OpenShift, Kubernetes, and Docker environment. We installed it for the Docker compose environment. We used Ansible AWX to generate the inventory of destination servers, distribute the SSH public key and install OpenJDK.

 

Issues identified during the review

Let me introduce a couple of issues identified during the OpenJDK review. It was not a compatibility or a performance issue of OpenJDK. It was related to LINE’s in-house developed applications because they used classes that were not included in OpenJDK.

In installing JCE

For Java development, JCE (Java Cryptography Extension) is installed additionally for stronger security. The jar file including provider information is added to JDK classpath to support higher bit encryption. As for versions prior to Java 8_151 of Oracle JDK, JCE should have been separately downloaded and installed. For later versions, JCE was included as a package and it could be activated with either of the following configurations.

  • Open the file, ${JAVA_HOME}/jre/lib/security/java.security, and uncomment crypto.policy=unlimited or if there is no such configuration, add this line.
  • Set configuration by adding this in your application code; Security.setProperty("crypto.policy", "unlimited");

As for OpenJDK, Java 8 version comes with JCE, activated by default. So, you don’t need to install or set configuration separately. However, as we had time constraints to review the source code in detail (a gap in the provider), we copied the JCE file like in the old way (through Ansible playbook).

In using JMX for collecting Tomcat WAS information

LINE’s internal monitoring system uses Java JMX to collect thread data of Tomcat, for which LINE implemented a dedicated class.

<Listener className="jp.naver.facility.jmxagent.MonitorLifecycleListener" .....

We needed to copy the jar file for performance data collection to continue to work. This was also done through Ansible playbook by copying the file at the time of OpenJDK installation.

In using JavaFX

Sun Microsystems, who developed JDK, decided to go open source on Java in November 2006. During this open sourcing process, some of the libraries that were not developed by Sun Microsystems were excluded from OpenJDK, and JavaFX was one of them. JavaFX is a client application platform for desktop to implement GUI or multimedia user interface. It is not used for ordinary web-based programs. However, if you need it, you can build and install OpenJFX.

As mentioned earlier, JavaFX is not usually used for web-based apps, but sometimes the JFX class is used unnecessarily. There are two ways to resolve this issue as the following. We used the second approach; removing JFX class from the code.

  • Build and install OpenJFX 
  • Remove the JFX class from the code, if possible.

Selecting distribution version

After conducting all possible tests and a series of discussions, we decided to adopt OpenJDK distributed by Red Hat. We deployed OpenJDK to LINE’s main services (part of instant messenger, authentication and channel), and it is working fine as of February 2019. We could have been simply lucky to select a perfectly clean version. Or, potential issues have not yet emerged because there was no chance. There is no single, right answer when it comes to selecting a version or a distribution. So, we chose what we think is the best for now.

Distribution version

We chose Red Hat’s OpenJDK for productivity and running costs, rather than for performance or compatibility. Performance was similar among tested distributions and we didn’t identify any compatibility issues. Please note that you could make a different decision, depending on your environment and resources.  

Productivity (Engineer friendly, stable working conditions)

  • Easy to deploy to multiple servers through distribution via yum repository 
  • Many of LINE’s servers are running on CentOS and maintained up to date with the latest versions such as CentOS 6 or 7.
  • Convenient automation with Ansible and yum module; bug fixes are done on CentOS.

Running costs (Human resources & licenses)

  • No additional costs as the internal systems are already running on CentOS
  • No need to internally make a separate build and maintain extra resources for TCK test, security, bug fix and version management
  • Java minor version upgrade can naturally catch up with future OS upgrades (stay with the specific version for stability)

Potential weakness – Zero day vulnerability

After reflecting to RHEL (Red Hat Enterprise Linux), the CentOS patch is not usually provided as a hot-fix, but reflected to the next version upgrade. In this case, the patch schedule could lag behind that of commercial JDK. If necessary, you might have to work on it yourself.

Here are commands to check if CVE code is applied:

#Checking the cve code for patch in the change log of RPM installed via yum
rpm -q --changelog java-1.8.0-openjdk | grep cve

#Checking available security patch via yum
yum list-security --security

#Check information on installed security patch
yum updateinfo list security all

#Patch a specific CVE 
yum update –cve CVE-2008-0947

Another potential problem is when there is an immediate security issue, but it is rarely the case. As you can see it below, there was only one case of CVE (Common Vulnerabilities and Exposures) in 2015 for OpenJDK. Of course, there are ongoing security patches for identified issues.

If a security issue is serious enough to be managed with the CVE Code, patches are likely to be provided quickly. Generally, JVM (WAS) does not provide service directly to outside (compared to web servers).

Considerations

Many tend to stay with the first JDK installed at the time of opening new service for years. Personally, I think it is always wise to reflect patches during OS upgrade or yum update and check if they work.

 

Things to note

If you are currently using Oracle JDK and don’t plan to sign up for subscription, you need to pay attention to a few things. I’m not an expert on subscription and licenses so I’ll just share a quick summary of what I know below.

Server

  • You must not update to versions later than Java 8 Public update (8u202) and Java 11 without license from January 2019.
  • You cannot download without OTN (Oracle Technology Network, Oracle’s service account). Even if you find a way to download, you will be in breach of Oracle license policy when you further update to a later version.

PC/Notebook

  • Same constraints apply as the servers.
  • As of February 2019, when you install Java8 update202, you will see a message, “Corporate users will be impacted with the April 2019 update.”
  • If Oracle JDK Update agent (auto update) is activated, you need to deactivate it or take necessary measures not to go for update.
    One option is to delete the current JDK and install HotSpot OpenJDK for Mac/Windows from AdoptOpenJDK.
  • Oracle will provide Java 8 updates for individual users only until 2020 (FYI).
  • PCs or laptops (regardless of personal ownership) used at offices are considered to be commercial, which can lead to a breach of the license policy.

Closing

I’ve finally come to an end of recap of my experience with the OpenJDK TF from October 2018 to January 2019. We invested a lot of time and effort in identifying the most appropriate test and verification methodology and actually conducting tests. We were lucky we didn’t experience any serious issues with OpenJDK. Like I mentioned in the beginning, OpenJDK is just one of numerous open source software we use everyday. It is only natural to have concerns before migration as it can have a direct impact on the code. Given that we already use many open source software such as Hadoop for large-scale data storage, Redis for Object Cache and Maria DB as RDB  for more critical infrastructure, there is no reason to stop from converting to open source software for JVM (JDK), which is much smaller in scale, down to tens or hundreds. Of course, it is very important to conduct the as-is analysis and thorough testing to prevent possible issues before making a final decision.

I hope my posting can be a little help for those of you who are thinking about moving to OpenJDK. Thank you.