2016 ScalingMemoryAugmentedNeuralNet

• (Rae et al., 2016) ⇒ Jack W Rae, Jonathan J Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, and Timothy P Lillicrap. (2016). “Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes.” In: Proceedings of Advances in Neural Information Processing Systems 29 (NIPS 2016). ISBN:978-1-5108-3881-9. e-print arXiv:1610.09027

Subject Headings: Sparse Access Memory Neural Network; Memory-Augmented Neural Network

Cited By

• http://scholar.google.com/scholar?q=%222016%22+Scaling+Memory-Augmented+Neural+Networks+with+Sparse+Reads+and+Writes
• http://dl.acm.org/citation.cfm?id=3157382.3157503&preflayout=flat#citedby
• http://papers.nips.cc/paper/6297-scaling-memory-augmented-neural-networks-with-sparse-reads-and-writes

Quotes

Abstract

Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows --- limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs 1, 000× faster and with 3, 000× less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring 100,000s of time steps and memories. As well, we show how our approach can beadapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.

Figures

.

---

---

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2016 ScalingMemoryAugmentedNeuralNet	Alex Graves Greg Wayne Ivo Danihelka Tim Harley Andrew Senior Jack W Rae Jonathan J Hunt Timothy P Lillicrap			Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes						2016