Before we begin
Hello, we are Jongkyu Lim and Joonghoon Shin responsible for processing of ads data and development of ads platform in LINE. In this blog post, I’d like to talk about BigDB: a big data processing pipeline for LINE ads. I’ll go into detail about how BigDB came to be, what BigDB is, how it’s structured, what it does, and what use cases it has.
How BigDB came to be
As we gather data from various services that are growing in size, the size of the data grows larger as well, making it difficult to properly use them for analysis. Up until now, we’ve used several open solutions that support big data to tackle this problem. While using several of these open solutions that have different strengths and weaknesses, we felt the need for a more simple and standardized way to collect, process, and look up data. And that is where the idea for BigDB began. Below are some of the key features we had in mind for BigDB.
What is BigDB
BigDB is a big data processing pipeline for LINE ads. It can collect, reprocess, and look up data. LINE ads are analyzed in two ways: The first is in real-time whenever an ad is shown to a user. The second is batch analysis, where events are collected and analyzed after a set time, such as hourly or daily. BigDB fluidly provides the data used for analysis. Sometimes combining time series and static data if necessary.