Spark

Troubleshooting Data Engineering Software

Hello, I’m Keiji Yoshida. I currently work as a data engineer at LINE Data Labs. In this posting, I’d like to share a few cases on data engineering software troubleshooting: Apache Hadoop YARN ResourceManager failover Apache Hadoop HDFS NameNode failover Apache Zeppelin Notebook scheduler issue Apache Zeppelin deadlock issue Apache Spark SQL performance issue Troubleshooting failover issue in Apache Hadoop […]

Analyzing Large Amounts of Security Data With Spark, Mesos, Zeppelin, and HDFS

Hello, we are WJ and KH and we are in charge of game security development at LINE.

A massive number of players play LINE Games at any given time, and that is why quick issue analysis and response can be difficult. With the large increase in LINE Game players and users from abroad, many instances of abuse (profiting through unauthorized actions) are being observed. Abusing the system affects other players that are playing the game by the rules, while also directly affecting the game itself. Responding to these abuses as quickly as possible is crucial to keeping the game stable, and to protect players that are playing the game as intended.