Root Mean Square Propagation Algorithm (RMSprop)

A Root Mean Square Propagation Algorithm (RMSprop) is a Gradient Descent-based Learning Algorithm that combines Adagrad and Adadelta methods.

• AKA: RMSProp, RMSprop, RMSProp Optimizer, RMSProp Algorithm.
• Example(s):
  • torch.optim.RMSprop,
  • chainer.optimizers.RMSprop,
  • chainer.optimizers.RMSpropGraves,
  • tflearn.optimizers.RMSProp,
  • RMSPROP from CRAN gradDescent Repository [1],
  • RMSProp in Deeplearning4j,
• Counter-Example(s):
  • an Accelerated Gradient Descent (AGD),
  • an Adaptive Gradient Algorithm (AdaGrad),
  • an Adaptive Learning Rate Algorithm (AdaDelta),
  • an Adaptive Moment Estimation Algorithm (Adam),
  • a Mini-Batch Gradient Descent Algorithm (MBGD).
  • a Momentum Gradient Descent (MGD),
  • a SGD-based Algorithm such as:
    • a Kalman-based Stochastic Gradient Descent (kSGD),
    • a Mini-Batch Stochastic Gradient Descent Algorithm (MBSGD),
    • a Stochastic Average Gradient (SAG),
    • a Semi-Stochastic Gradient Descent (SSGD),
    • a Stochastic Recursive Gradient Algorithm (SARAH),
    • a Stochastic Variance Reduce Gradient (SVRG).
• See: Stochastic Optimization, Convex Optimization, Learning Rate, Gradient Descent, Outer Product, Hadamard Matrix Product, Euclidean Norm, Proximal Function, Rprop, Stochastic Gradient Descent.

---

---

References

2018a

• (Wikipedia, 2018) ⇒ https://en.wikipedia.org/wiki/Stochastic_gradient_descent#RMSProp Retrieved:2018-4-29.
  • RMSProp (for Root Mean Square Propagation) is also a method in which the learning rate is adapted for each of the parameters. The idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight. So, first the running average is calculated in terms of means square,

    [math]\displaystyle{ v(w,t):=\gamma v(w,t-1)+(1-\gamma)(\nabla Q_i(w))^2 }[/math]

    where, [math]\displaystyle{ \gamma }[/math] is the forgetting factor. And the parameters are updated as,

    [math]\displaystyle{ w:=w-\frac{\eta}{\sqrt{v(w,t)}}\nabla Q_i(w) }[/math]

    RMSProp has shown excellent adaptation of learning rate in different applications. RMSProp can be seen as a generalization of Rprop and is capable to work with mini-batches as well opposed to only full-batches.

2018b

• (Wijaya et al., 2018) ⇒ Galih Praja Wijaya, Dendi Handian, Imam Fachmi Nasrulloh, Lala Septem Riza, Rani Megasari, Enjun Junaeti (2018), "gradDescent: Gradient Descent for Regression Tasks", "Reference manual (PDF).
  • QUOTE: An implementation of various learning algorithms based on Gradient Descent for dealing with regression tasks. The variants of gradient descent algorithm are: Mini-Batch Gradient Descent (MBGD), which is an optimization to use training data partially to reduce the computation load. Stochastic Gradient Descent (SGD), which is an optimization to use a random data in learning to reduce the computation load drastically. Stochastic Average Gradient (SAG), which is a SGD-based algorithm to minimize stochastic step to average. Momentum Gradient Descent (MGD), which is an optimization to speed-up gradient descent learning. Accelerated Gradient Descent (AGD), which is an optimization to accelerate gradient descent learning. Adagrad, which is a gradient-descent-based algorithm that accumulate previous cost to do adaptive learning. Adadelta, which is a gradient-descent-based algorithm that use hessian approximation to do adaptive learning. RMSprop, which is a gradient-descent-based algorithm that combine Adagrad and Adadelta adaptive learning ability. Adam, which is a gradient-descent-based algorithm that mean and variance moment to do adaptive learning. Stochastic Variance Reduce Gradient (SVRG), which is an optimization SGD-based algorithm to accelerates the process toward converging by reducing the gradient. Semi Stochastic Gradient Descent (SSGD),which is a SGD-based algorithm that combine GD and SGD to accelerates the process toward converging by choosing one of the gradients at a time. Stochastic Recursive Gradient Algorithm (SARAH), which is an optimization algorithm similarly SVRG to accelerates the process toward converging by accumulated stochastic information. Stochastic Recursive Gradient Algorithm+ (SARAHPlus), which is a SARAH practical variant algorithm to accelerates the process toward converging provides a possibility of earlier termination.

2018c

• (DL4J, 2018) ⇒ https://deeplearning4j.org/updater#rmsprop Retrieved: 2018-04-29.
  • QUOTE: The only difference between RMSProp and Adagrad is that the g_t term is calculated by exponentially decaying the average and not the sum of gradients.

    Here g_t is called the second order moment of [math]\displaystyle{ \delta L }[/math]. Additionally, a first-order moment m_t can also be introduced.

    Adding momentum, as in the first case…

    …and finally collecting a new theta as we did in the first example.

    • RMSPropUpdater in Deeplearning4j

2018d

• (Redii et al.) ⇒ Reddi, S. J., Kale, S., & Kumar, S. (2018, February). "On the convergence of adam and beyond. In International Conference on Learning Representations (PDF)" [2].
  • ABSTRACT: Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with “long-term memory of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.

2015

• (Misra, 2015) ⇒ Ishan Misra (2015)."Optimization for Deep Networks" (PDF)
  • QUOTE: RMSProp = Rprop + SGD.
    • Tieleman & Hinton et al., 2012 (Coursera slide 29, Lecture 6)
    • Scale updates similarly across mini-batches,
    • Scale by decaying average of squared gradient,
      • Rather than the sum of squared gradients in AdaGrad.

        [math]\displaystyle{ r_t=(1-\gamma)f'(\theta)^2+\gamma r_{t-1} }[/math]

        [math]\displaystyle{ v_{t+1}=\frac{\alpha}{\sqrt{r_t}f'(\theta_t)} }[/math],

        [math]\displaystyle{ \theta_{t+1}=\theta_t-v_{t+1} }[/math]

2013

• (Graves, 2013) ⇒ Graves, A. (2013). "Generating sequences with recurrent neural networks"(PDF). arXiv preprint arXiv:1308.0850.
  • ABSTRACT: This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

---