MapReduce Software Framework

A MapReduce Software Framework is a distributed computing data processing framework that can implement map/reduce-based systems (which process map/reduce jobs).

• Context:
  • It can (typically) contain Distributed File System, and a Map-Reduce Component.
• Example(s):
  • Apache Hadoop.
  • Google's MR Framework.
  • …
• Counter-Example(s):
  • an Apache Spark Framework.
  • an Distributed File System.
• See: Hadoop YARN, BIGTABLE.

---

---

References

2014

• Michael Stonebraker. http://cacm.acm.org/magazines/2015/1/181612-a-valuable-lesson-and-whither-hadoop/fulltext

2010

• (Dean & Ghemawat, 2010) ⇒ Jeffrey Dean, and Sanjay Ghemawat. (2010). “MapReduce: A Flexible Data Processing Tool.” In: Communications of the ACM Journal, 53(1). doi:10.1145/1629175.1629198
• (Lin & Dyer, 2010) ⇒ Jimmy Lin, and Chris Dyer. (2010). “Data-Intensive Text Processing with MapReduce." Morgan Claypool Publishers doi:10.2200/S00274ED1V01Y201006HLT007 ISBN:1608453421

2009

• http://en.wikipedia.org/wiki/MapReduce
  • MapReduce is a patented software framework introduced by Google to support distributed computing on large data sets on clusters of computers.
  • The framework is inspired by map and reduce functions commonly used in functional programming, although their purpose in the MapReduce framework is not the same as their original forms.
  • MapReduce libraries have been written in C++, C#, Erlang, Java, Python, Ruby, F#, R and other programming languages.

2008

• (Dean & Ghemawat, 2008) ⇒ Jeffrey Dean, and Sanjay Ghemawat. (2008). “MapReduce: Simplified Data Processing on Large Clusters.” Communications of the ACM 51, no. 1

2006

• (Chu et al., 2006) ⇒ Cheng-Tao Chu, Sang Kyun Kim, Yi-An Lin, YuanYuan Yu, Gary Bradski, Andrew Y. Ng, and Kunle Olukotun. (2006). “Map-Reduce for Machine Learning on Multicore.” In: Proceedings of the 19th International Conference on Neural Information Processing Systems.
  • QUOTE: ... We adapt Google's map-reduce [7] paradigm to demonstrate this parallel speed up technique on a variety of learning algorithms including locally weighted linear regression (LWLR), k-means, logistic regression (LR), naive Bayes (NB), SVM, ICA, PCA, gaussian discriminant analysis (GDA), EM, and backpropagation (NN). ...

2004

• (Dean & Ghemawat, 2004a) ⇒ Jeffrey Dean, and Sanjay Ghemawat. (2004). “System and method for efficient large-scale data processing." US Patent 7,650,331
• (Dean & Ghemawat, 2004b) ⇒ Jeffrey Dean, and Sanjay Ghemawat. (2004). “MapReduce: Simplified Data Processing on Large Clusters.” In: Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation (OSDI 2004).

---