Hello, everyone. I'm Nao, a developer of LINE's Messaging Server.
In 2014 we introduced a new development and release process for LINE Server (Blog post in Japanese only). Since then, the development environment and code has changed. In this post, I'd like to introduce the points that have been improved.
LINE's Messaging Platform consists of many Server Applications. One of them, Messaging Server, is a platform responsible for messaging functions, and we're working on multiple projects in parallel with developers from other teams.
Previous Development and Release Process
When working on multiple projects at the same time, it's easy to get confused about what is currently being done in each project. In order to improve this problem as much as possible, we have been experimenting with rules and policies to minimize the time and effort required for collaborative development. The result of this process is the previously mentioned development process for LINE's Messaging Server, which is shown below.
The previous development process can be characterized by the fact that feature development is done by creating a feature branch on the develop branch, and release is done by creating a release branch from the master branch, then cherry-picking and applying the changes to the master branch.
The testing process includes Unit Tests, beta QA (Developer QA, QA team testing, Integration Test), RC QA, and various other testing methods.
- Developers always implement Unit Test as a set.
- After merging to the develop branch, we test with other components in a beta environment.
- There is also Integration Test, which automatically runs hourly tests against the beta server. We also add Integration Tests along with feature development. Integration Test makes it easier to notice regressions when other components change or unintended changes occur.
- The QA team regularly conducts regression tests using the LINE client.
- Once the Feature Test is complete, we merge the changes into the master branch and perform RC QA testing with other changes released at the same time in the RC environment.
- When deploying to the Release environment, we first release to the server in a canary group and then apply it to the entire environment.
Background
The Messaging Server code has a long history within LINE, and as time went by, we started noticing things that we wanted to improve. Things that were difficult to improve before were now possible with the progress we've seen in Microservices.
Let's consider the case below. On the develop branch, Feature A was applied first, then Feature B. However, we decided to release Feature B first. At that time, a cherry-pick conflict may occur and you may have to resolve the conflict before releasing. In this case, we test on the Release branch.
There's a problem here.
- The part where the conflict caused by cherry-pick is resolved may differ from the develop branch, and some code may differ between the develop/master branch.
- Since we're developing on unreleased code, we will develop/test based on code that includes unreleased features.
In order to eliminate the difference between the develop branch and the master branch, we regularly perform a "branch reset" to re-create the develop branch from master branch, but we tried to improve it so that it can be easily tested with a little more visualization/automation.
Mission
- Run tests in isolation from other unreleased code during development
- Visualize the progress of development by grasping the features released from the develop branch and the features that haven't been released
Goal
We made the following improvements to achieve the above mission:
- Feature branch is created from master branch
- Change from cherry-pick-based workflow to merge-based workflow to sync git history between master and develop branches
- Provide an isolated test environment using containers and other technology
The final image looks like this. Each feature branch can be created from the master branch, then tested and released in an isolated environment.
In the previous development process, it was possible to test in a local environment to some extent, but with Microservices, testing in a local environment becomes more difficult.
For testing in the local environment, the Messaging Server is launched in the local environment, and it's connected to the alpha and beta environments of other components and tested in combination with those environments. Integration Test can also be run and tested in Local mode.
New branch management workflow
Before I introduce the new development process, let's first compare the well-known workflows Gitflow and GitHub flow.
With Gitflow, release is done by creating a release branch from the develop branch and merging the release branch into the master branch. In GitHub flow, there is no develop branch, and it's a simple workflow that manages only the master branch. Many projects are developed in GitHub flow within the company.
Messaging Server has many independent feature developments underway. Messaging Server provides a beta environment for testing in conjunction with other components. The develop branch exists for this beta environment. Since Messaging Server has a daily release and the release cycle is fast, it isn't possible to just merge the develop branch into the master branch and release it. So Gitflow is not suitable for our release cycle. By leaving the develop branch and then developing on the master branch like GitHub flow, it's possible to develop, test, and release independently from other features.
The basic concept looks like the diagram below. Development and testing is done on the master branch, and beta QA and RC QA is done the same as before. When releasing, we merge the feature branch directly into the master branch. There is something called Back Merge here, which I'll explain later.
Here is one complex example. This is a case where a conflict occurred when merging the feature branch into the develop branch. In this case, we create a conflict-resolution branch from the develop branch, and then merge the feature branch into a conflict-resolution branch with conflict resolution. After that, we merge this conflict-resolution branch into the develop branch. The release is the same as the basic workflow, where we merge the feature branch into the master branch.
Benefits of this workflow
- Using the git command makes it easier to automate and visualize various things.
- You can visualize the progress of development to see which feature branch on the develop branch has been released or not
- Dependencies between feature branches can also be visualized
- If you test on unreleased code and release your feature in advance, the code path you took when testing will change. Therefore, testing on the code of the release environment increases test reliability in the development stage.
Difficulties
During the transition to a new workflow, it was difficult to explain the new workflow to all members of the various teams involved, which resulted in the previous workflow often being favored.
Even though we converted to using Microservices to some extent and divided the server into multiple server components, conflicts were likely to occur because many developers were involved. Our largest pain point was the cost of creating the conflict-resolution branch.
In order to prevent such misoperations and reduce conflict cases, improvement of the test environment and tools was required, which resulted in additional maintenance costs.
Tools to support the new development process
There are various tools we made, but I'll introduce PR Checker, Back Merge, and Weekly Reporter among them.
PR Checker
This is a tool to prevent the branch from being created by the wrong workflow in a pull request. This tool eliminates the possibility of incorrectly created branches merging into develop branches.
Feature branch
Verifies that the feature branch was created from the master branch.
Conflict-resolution branch
Detects when a new branch is created from the develop branch and the feature branch is merged into it.
Wrong branch
Displays a warning if a feature was developed on the develop branch.
Back Merge
Let's talk about Back Merge.
To understand why Back Merge is necessary, it's necessary to understand commit types.
There are two types of commit types: merge commit and non-merge commit. Merge commit and non-merge commit are determined by the number of parent commits.
In the figure below, c1, c2, c3, f1, and f2 are non-merge commits with only one parent commit, while m1 is a merge commit created when merging multiple branches and has more than one parent commit.
Merge commits made when releasing to the master branch also need to be synced to the develop branch.
Back Merge is the operation of merging a merge commit without differences into the develop branch when merging into the master branch.
Weekly Reporter
Weekly Reporter provides development progress on the develop branch.
The original purpose of Weekly Reporter was to reduce the number of conflicts. Conflicts can be caused by a branch that has been merged with a develop branch but has not been released for a long time. The reason for not releasing a branch for a long time may be that it's not scheduled to be released and can't be tested in the local environment.
In order to visualize the progress of development and the cause of conflicts, GitHub pages is used to provide the following functions.
- Unreleased branch
- Branches created in the wrong workflow
- Conflict-resolution branch
- Branches that caused the conflict in the Conflict-resolution branch
Unreleased branch
Shows feature branches that have been merged into the develop branch but not released in chronological order. This Weekly Reporter function is also used for Branch Resetter to re-create a new develop branch from the master branch. When re-creating, it tries merging in order from the oldest feature, and also displays whether it can be merged without conflicts. If there is a conflict, it also shows dependencies and branches. Since Branch Resetter automatically merges things that can be auto-merged, it's now possible to automate branch resets that were previously done manually.
Branches created in the wrong workflow
Shows branches that were created by mistake among the branches that were merged into the develop branch. If you find an erroneously created branch, you have to revert it. Now, thanks to PR Checker, the wrong branch is no longer created.
Conflict-resolution branches
Displays how often conflicts occur and in what cases.
Branches caused conflicts of conflict-resolution branches
Displays branches that caused the conflict-resolution branch to be created in descending order of impact. You can check why it has not been released and seek improvements.
After the introduction of the new development workflow
At first, the learning cost was high because the workflow was unfamiliar to us, but now we can easily see the progress of forgotten branches that were never released. When a conflict occurs, the following actions are taken among developers.
- There is a tendency to discuss and decide in advance when to release the code and how to test it among those who are conflicted. In a sense, this speeds up the release cycle.
- Feature flags are used to disable features and gradually pre-release them.
- Refactoring the code to a structure that is less likely to cause conflicts.
In some cases, it wasn't possible to test in a local environment due to various restrictions such as environment specifications. Therefore, it was necessary to test in the beta environment. In order to enable beta testing without merging into the beta environment, a task force was established with the cooperation of developers from various projects to improve the testing environment. We used container technology to set up a server equivalent to the beta in a simple container. We call this Beta Container.
We will introduce how we built the Beta Container in another article.
Lastly
LINE has many server applications and branch management methods vary depending on the purpose of each project. In this article, we have introduced improvements to the branch management method for the Messaging Server development workflow.
There are still other issues to be addressed. Merge commits from the develop branch need to be synced to the master branch. Therefore, we're looking into various ways to do this, such as introducing the concept of Forward-Merge, which syncs merge commits in the develop branch to the master branch or automatically runs Branch Resetter on a periodic basis whenever all unreleased features can be auto-merged. We also have multiple beta servers running in-house. A callback from another team's server application may call the server specified in advance. we're continuously making improvements such as dynamically changing the server to call and expanding the range of tests.
Our development process is constantly improving. I'd like to take this opportunity to say thank you to all the teams involved. Thank you very much.