2013 SharkSQLandRichAnalyticsatScale

(Xin et al., 2013) ⇒ Reynold S. Xin, Josh Rosen, Matei Zaharia, Michael J. Franklin, Scott Shenker, and Ion Stoica. (2013). “Shark: SQL and Rich Analytics at Scale.” In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. ISBN:978-1-4503-2037-5 doi:10.1145/2463676.2465288

Subject Headings: Shark System, Spark SQL.

Notes

Cited By

2015

(Armbrust et al., 2015) ⇒ Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, and Matei Zaharia. (2015). “Spark SQL: Relational Data Processing in Spark.” In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ISBN:978-1-4503-2758-9 doi:10.1145/2723372.2742797
- QUOTE: Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark,

Quotes

Abstract

Shark is a new data analysis system that marries query processing with complex analytics on large clusters. It leverages a novel distributed memory abstraction to provide a unified engine that can run SQL queries and sophisticated analytics functions (e.g. iterative machine learning) at scale, and efficiently recovers from failures mid-query. This allows Shark to run SQL queries up to 100X faster than Apache Hive, and machine learning programs more than 100X faster than Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, and the fine-grained fault tolerance properties that such engine provides. It extends such an engine in several ways, including column-oriented in-memory storage and dynamic mid-query replanning, to effectively execute SQL. The result is a system that matches the speedups reported for MPP analytic databases over MapReduce, while offering fault tolerance properties and complex analytics capabilities that they lack.

References

1. https://github.com/cloudera/impala.
2. http://hadoop.apache.org/.
3. http://aws.amazon.com/elasticmapreduce/.
4. Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, Avi Silberschatz, Alexander Rasin, HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads, Proceedings of the VLDB Endowment, v.2 n.1, August 2009
5. Sameer Agarwal, Srikanth Kandula, Nicolas Bruno, Ming-Chuan Wu, Ion Stoica, Jingren Zhou, Re-optimizing Data-parallel Computing, Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, April 25-27, 2012, San Jose, CA
6. Ganesh Ananthanarayanan, Ali Ghodsi, Andrew Wang, Dhruba Borthakur, Srikanth Kandula, Scott Shenker, Ion Stoica, PACMan: Coordinated Memory Caching for Parallel Jobs, Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, April 25-27, 2012, San Jose, CA
7. Ron Avnur, Joseph M. Hellerstein, Eddies: Continuously Adaptive Query Processing, Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, p.261-272, May 15-18, 2000, Dallas, Texas, USA doi:10.1145/342009.335420
8. Shivnath Babu, Towards Automatic Optimization of MapReduce Programs, Proceedings of the 1st ACM Symposium on Cloud Computing, June 10-11, 2010, Indianapolis, Indiana, USA doi:10.1145/1807128.1807150
9. Alexander Behm, Vinayak R. Borkar, Michael J. Carey, Raman Grover, Chen Li, Nicola Onose, Rares Vernica, Alin Deutsch, Yannis Papakonstantinou, Vassilis J. Tsotras, ASTERIX: Towards a Scalable, Semistructured Data Platform for Evolving-world Models, Distributed and Parallel Databases, v.29 n.3, p.185-216, June 2011 doi:10.1007/s10619-011-7082-y
10. Vinayak Borkar, Michael Carey, Raman Grover, Nicola Onose, Rares Vernica, Hyracks: A Flexible and Extensible Foundation for Data-intensive Computing, Proceedings of the 2011 IEEE 27th International Conference on Data Engineering, p.1151-1162, April 11-16, 2011 doi:10.1109/ICDE.2011.5767921
11. Yingyi Bu, Bill Howe, Magdalena Balazinska, Michael D. Ernst, HaLoop: Efficient Iterative Data Processing on Large Clusters, Proceedings of the VLDB Endowment, v.3 n.1-2, September 2010
12. Ronnie Chaiken, Bob Jenkins, Per-Åke Larson, Bill Ramsey, Darren Shakib, Simon Weaver, Jingren Zhou, SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets, Proceedings of the VLDB Endowment, v.1 n.2, August 2008
13. B. Chattopadhyay,, Et Al. Tenzing a Sql Implementation on the Mapreduce Framework. PVLDB, 4(12):1318--1327, 2011.
14. Songting Chen, Cheetah: A High Performance, Custom Data Warehouse on Top of MapReduce, Proceedings of the VLDB Endowment, v.3 n.1-2, September 2010
15. C. Chu Et Al. Map-Reduce for Machine Learning on Multicore. Advances in Neural Information Processing Systems, 19:281, 2007.
16. Jeffrey Cohen, Brian Dolan, Mark Dunlap, Joseph M. Hellerstein, Caleb Welton, MAD Skills: New Analysis Practices for Big Data, Proceedings of the VLDB Endowment, v.2 n.2, August 2009
17. Jeffrey Dean, Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation, p.10-10, December 06-08, 2004, San Francisco, CA
18. Xixuan Feng, Arun Kumar, Benjamin Recht, Christopher Ré, Towards a Unified Architecture for in-RDBMS Analytics, Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, May 20-24, 2012, Scottsdale, Arizona, USA doi:10.1145/2213836.2213874
19. B. Guffler Et Al. Handling Data Skew in Mapreduce. In CLOSER'11.
20. Alexander Hall, Olaf Bachmann, Robert Büssow, Silviu Gănceanu, Marc Nunkesser, Processing a Trillion Cells Per Mouse Click, Proceedings of the VLDB Endowment, v.5 n.11, p.1436-1446, July 2012
21. Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica, Mesos: A Platform for Fine-grained Resource Sharing in the Data Center, Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, March 30-April 01, 2011, Boston, MA
22. Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly, Dryad: Distributed Data-parallel Programs from Sequential Building Blocks, ACM SIGOPS Operating Systems Review, v.41 n.3, June 2007 doi:10.1145/1272998.1273005
23. Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, Andrew Goldberg, Quincy: Fair Scheduling for Distributed Computing Clusters, Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, October 11-14, 2009, Big Sky, Montana, USA doi:10.1145/1629575.1629601
24. Michael Isard, Yuan Yu, Distributed Data-parallel Computing Using a High-level Programming Language, Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, June 29-July 02, 2009, Providence, Rhode Island, USA doi:10.1145/1559845.1559962
25. Navin Kabra, David J. DeWitt, Efficient Mid-query Re-optimization of Sub-optimal Query Execution Plans, ACM SIGMOD Record, v.27 n.2, p.106-117, June 1998 doi:10.1145/276305.276315
26. YongChul Kwon, Magdalena Balazinska, Bill Howe, Jerome Rolia, SkewTune: Mitigating Skew in Mapreduce Applications, Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, May 20-24, 2012, Scottsdale, Arizona, USA doi:10.1145/2213836.2213840
27. Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, Joseph M. Hellerstein, Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud, Proceedings of the VLDB Endowment, v.5 n.8, p.716-727, April 2012
28. Grzegorz Malewicz, Matthew H. Austern, Aart J.C Bik, James C. Dehnert, Ilan Horn, Naty Leiser, Grzegorz Czajkowski, Pregel: A System for Large-scale Graph Processing, Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, June 06-10, 2010, Indianapolis, Indiana, USA doi:10.1145/1807167.1807184
29. Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis, Dremel: Interactive Analysis of Web-scale Datasets, Proceedings of the VLDB Endowment, v.3 n.1-2, September 2010
30. Kay Ousterhout, Aurojit Panda, Joshua Rosen, Shivaram Venkataraman, Reynold Xin, Sylvia Ratnasamy, Scott Shenker, Ion Stoica, The Case for Tiny Tasks in Compute Clusters, Proceedings of the 14th USENIX Conference on Hot Topics in Operating Systems, p.14-14, May 13-15, 2013, Santa Ana Pueblo, New Mexcio
31. Andrew Pavlo, Erik Paulson, Alexander Rasin, Daniel J. Abadi, David J. DeWitt, Samuel Madden, Michael Stonebraker, A Comparison of Approaches to Large-scale Data Analysis, Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, June 29-July 02, 2009, Providence, Rhode Island, USA doi:10.1145/1559845.1559865
32. Mike Stonebraker, Daniel J. Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O'Neil, Pat O'Neil, Alex Rasin, Nga Tran, Stan Zdonik, C-store: A Column-oriented DBMS, Proceedings of the 31st International Conference on Very Large Data Bases, August 30-September 02, 2005, Trondheim, Norway
33. Michael Stonebraker, Daniel Abadi, David J. DeWitt, Sam Madden, Erik Paulson, Andrew Pavlo, Alexander Rasin, MapReduce and Parallel DBMSs: Friends Or Foes?, Communications of the ACM, v.53 n.1, January 2010 doi:10.1145/1629175.1629197
34. A. Thusoo Et Al. Hive-a Petabyte Scale Data Warehouse Using Hadoop. In ICDE, 2010.
35. Transaction Processing Performance Council. TPC BENCHMARK H.
36. Tolga Urhan, Michael J. Franklin, Laurent Amsaleg, Cost-based Query Scrambling for Initial Delays, ACM SIGMOD Record, v.27 n.2, p.130-141, June 1998 doi:10.1145/276305.276317
37. C. Yang Et Al. Osprey: Implementing Mapreduce-style Fault Tolerance in a Shared-nothing Distributed Database. In ICDE, 2010.
38. Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, Ion Stoica, Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling, Proceedings of the 5th European Conference on Computer Systems, April 13-16, 2010, Paris, France doi:10.1145/1755913.1755940
39. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica, Resilient Distributed Datasets: A Fault-tolerant Abstraction for in-memory Cluster Computing, Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, April 25-27, 2012, San Jose, CA

}};

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2013 SharkSQLandRichAnalyticsatScale	Ion Stoica Matei Zaharia Michael J. Franklin Scott Shenker Reynold S. Xin Josh Rosen			Shark: SQL and Rich Analytics at Scale				10.1145/2463676.2465288		2013

2013 SharkSQLandRichAnalyticsatScale

Notes

Cited By

2015

Quotes

Abstract

References

Navigation menu

Search