2015 SparkSQLRelationalDataProcessin

• (Armbrust et al., 2015) ⇒ Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, and Matei Zaharia. (2015). “Spark SQL: Relational Data Processing in Spark.” In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ISBN:978-1-4503-2758-9 doi:10.1145/2723372.2742797

Subject Headings: Spark SQL Module, Spark SQL,

Cited By

• http://scholar.google.com/scholar?q=%222015%22+Spark+SQL%3A+Relational+Data+Processing+in+Spark
• http://dl.acm.org/citation.cfm?id=2723372.2742797&preflayout=flat#citedby

Quotes

Abstract

Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g. declarative queries and optimized storage), and lets SQL users call complex analytics libraries in Spark (e.g. machine learning). Compared to previous systems, Spark SQL makes two main additions. First, it offers much tighter integration between relational and procedural processing, through a declarative DataFrame API that integrates with procedural Spark code. Second, it includes a highly extensible optimizer, Catalyst, built using features of the Scala programming language, that makes it easy to add composable rules, control code generation, and define extension points. Using Catalyst, we have built a variety of features (e.g. schema inference for JSON, machine learning types, and query federation to external databases) tailored for the complex needs of modern data analysis. We see Spark SQL as an evolution of both SQL-on-Spark and of Spark itself, offering richer APIs and optimizations while keeping the benefits of the Spark programming model.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2015 SparkSQLRelationalDataProcessin	Yin Huai Matei Zaharia Michael J. Franklin Reynold S. Xin Michael Armbrust Cheng Lian Davies Liu Joseph K. Bradley Xiangrui Meng Tomer Kaftan Ali Ghodsi			Spark SQL: Relational Data Processing in Spark				10.1145/2723372.2742797		2015