Matei Zaharia & Reynold Xin As big data becomes a concern for more and more organizations, there is a need for both faster tools to process it and easier-to-use APIs. Spark is a Hadoop-compatible cluster computing engine that addresses these needs through (1) in-memory computing primitives that let it run 100x faster than Hadoop and (2) a concise and high-level Scala API that can be used both in standalone programs and interactively from the Scala shell. Increasingly, Spark is also being used to power a stack of higher-level computing frameworks, including Shark, a port of the Hive SQL engine, and GraphX, a graph computing package that can describe algorithms like PageRank in a few lines of code. In this talk, we'll introduce the Spark ecosystem, focusing on Spark's native Scala API and GraphX. We'll also cover use cases from Spark's open source community, which has grown significantly since we released the project in 2010 -- in the past year, 15 companies have contributed code to Spark.

No outline is available