Large Deep Neural Network

A Large Deep Neural Network is a deep neural network with over one billion NNet parameters.

• Example(s):
  • a Large Deep Neural Language Model, such as Turing-NLG.
• See: Small DNN.

---

---

References

2019

• (Rajbhandari et al., 2019) ⇒ Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. (2019). “ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.” In: arXiv preprint arXiv:1910.02054.
  • QUOTE: ... Training large DL models with billions and potentially trillions of parameters is challenging. Existing solutions exhibit fundamental limitations to obtain both memory and scaling (computation / communication) efficiency together. Data parallelism does not help reduce memory footprint per device: a model with 1.5 billion parameters or more runs out of memory. …

---