Apache Spark Cluster Instance

Context:
- It can (typically) run Spark Applications / Spark Jobs.
- It can (typically) be composed of Spark Cluster Manager (standalone, Mesos, YARN) and Spark Worker Nodes (which run tasks on an executor).
- It can be composed of Spark Driver Program and Spark Tasks (within Spark Executors).
- It can range from being a Long-Lived Spark Cluster / Permanent Spark Cluster to being a Short-Lived Spark Cluster / Temporary Spark Cluster.
- …
Example(s):
- the j-1TCRZQMPC9R64 AWS EMR-based Spark Cluster (based on AWS EMR).
- a DataBricks-based Spark Cluster.
- …
Counter-Example(s):
- a Scalding Cluster (for Scalding jobs).
- a MapReduce Cluster (such as a Hadoop cluster).
See: HDFS, Hadoop Master Node, Spark Context, Grid Computing.

References

Component	Type	Behaviour on Component Failure	Resilient
Driver	Process	Client Mode: The entire application is killed
Driver	Process	Cluster Mode with supervise: The Driver is restarted on a different Worker node
Master	Process	Single Master: The entire application is killed
Master	Process	Multi Master: A STANDBY master is elected ACTIVE
Worker Process	Process	All child processes (executor or driver) are also terminated and a new worker process is launched
Executor	Process	A new executor is launched by the Worker process
Receiver	Thread(s)	Same as Executor as they are long running tasks inside the Executor
Worker Node	Node	Worker, Executor and Driver processes run on Worker nodes and the behavior is same as killing them individually

http://en.wikipedia.org/wiki/Apache_Hadoop#Architecture
- … A small Hadoop cluster will include a single master and multiple worker nodes. The master node consists of a JobTracker, TaskTracker, NameNode, and DataNode. A slave or worker node acts as both a DataNode and TaskTracker, though it is possible to have data-only worker nodes, and compute-only worker nodes; these are normally only used in non-standard applications.