LINE Engineering

Total 1 Tagged in ’#Data-pipeline’
  • BigDB - an ad data pipeline for LINE
    Jongkyu Lim, Joonghoon Shin 2017.07.12

    Jongkyu Lim: Jongkyu Lim is responsible for processing ads data. Joonghoon Shin: Joonghoon Shin develops ads data platform in LINE.

    Before we begin

    Hello, we are Jongkyu Lim and Joonghoon Shin responsible for processing of ads data and development of ads platform in LINE. In this blog post, I'd like to talk about BigDB: a big data processing pipeline for LINE ads. I'll go into detail about how BigDB came to be, what BigDB is, how it's structured, what it does, and what use cases it has.

    How BigDB came to be

    As we gather data from various services that are growing in size, the size of the data grows larger as well, making it difficult to properly use them for analysis. Up until now, we've used several open solutions that support big data to tackle this problem. While using several of these open solutions that have different strengths and weaknesses, we felt the need for a more simple and standardized way to collect, process, and look up data. And that is where the idea for BigDB began. Below are some of the key features we had in mind for BigDB.

    Introducing BigDB

    What is BigDB

    BigDB is a big data processing pipeline for LINE ads. It can collect, reprocess, and look up data. LINE ads are analyzed in two ways: The first is in real-time whenever an ad is shown to a user. The second is batch analysis, where events are collected and analyzed after a set time, such as hourly or daily. BigDB fluidly provides the data used for analysis. Sometimes combining time series and static data if necessary.

    Big-Database Big-DB Data-pipeline

    Read More