2017 FractalNetUltraDeepNeuralNetwor

• (Larsson et al., 2017) ⇒ Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. (2017). “FractalNet: Ultra-Deep Neural Networks Without Residuals.” In: Proceedings of the 5th International Conference on Learning Representations (ICLR 2017) Conference Track.

Subject Headings: FractalNet; Self-Similarity Neural Network, Fractal.

Cited By

• Google Scholar: ~ 712 Citations.

Quotes

Abstract

We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.

BibTeX

@inproceedings{DBLP:conf/iclr/LarssonMS17,
  author    = {Gustav Larsson and
               Michael Maire and
               Gregory Shakhnarovich},
  title     = {FractalNet: Ultra-Deep Neural Networks without Residuals},
  booktitle = {Proceedings of the 5th International Conference on Learning Representations
               (ICLR 2017) Conference Track},
  publisher = {OpenReview.net},
  year      = {2017},
  url       = {https://openreview.net/forum?id=S1VaB4cex},
}

---

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2017 FractalNetUltraDeepNeuralNetwor	Michael Maire Gustav Larsson Gregory Shakhnarovich			FractalNet: Ultra-Deep Neural Networks Without Residuals						2017