View source for Adaptive Gradient (AdaGrad) Algorithm

An [[Adaptive Gradient (AdaGrad) Algorithm]] is a [[gradient descent-based learning algorithm]] with a [[learning rate]] per [[parameter]].
* <B>Context:</B>
** It was first developed by [[2011_AdaptiveSubgradientMethodsforOn|Duchi et al., (2011)]].
** …
* <B>Example(s):</B>
** an [[Adagrad Dual Averaging algorithm (AdagradDA)]], e.g. <code>tf.train.AdagradDAOptimizer</code> [https://www.tensorflow.org/api_docs/python/tf/train/AdagradDAOptimizer]
** a [[Proximal AdaGrad]], e.g. <code>tflearn.optimizers.ProximalAdaGrad</code>[http://tflearn.org/optimizers/#proximaladagrad]
** <code>tf.train.AdagradOptimizer</code>[https://www.tensorflow.org/api_docs/python/tf/train/AdagradOptimizer]
** <code>ADAGRAD</code> from [[CRAN]] gradDescent Repository [https://cran.r-project.org/web/packages/gradDescent].
** <code>chainer.optimizers.AdaGrad</code> [http://docs.chainer.org/en/stable/reference/generated/chainer.optimizers.AdaGrad.html], [https://github.com/chainer/chainer/blob/v4.0.0/chainer/optimizers/ada_grad.py#L65];
** <code>torch.optim.Adagrad</code> [http://pytorch.org/docs/master/optim.html#torch.optim.Adagrad],[http://pytorch.org/docs/master/_modules/torch/optim/adagrad.html#Adagrad],
** <code>tflearn.optimizers.AdaGrad</code> [http://tflearn.org/optimizers/#adagrad].
** …
* <B>Counter-Example(s):</B>
** an [[Accelerated Gradient Descent (AGD)]],
** an [[Adaptive Learning Rate Algorithm (AdaDelta)]],
** an [[Adaptive Moment Estimation Algorithm (Adam)]],
** a [[Mini-Batch Gradient Descent Algorithm (MBGD)]].
** a [[Momentum Gradient Descent (MGD)]],
** a [[Root Mean Square Propagation Algorithm (RMSprop)]],
** a [[SGD-based Algorithm]] such as:
*** a [[Kalman-based Stochastic Gradient Descent (kSGD)]],
*** a [[Mini-Batch Stochastic Gradient Descent Algorithm (MBSGD)]],
*** a [[Stochastic Average Gradient (SAG)]],
*** a [[Semi-Stochastic Gradient Descent (SSGD)]],
*** a [[Stochastic Recursive Gradient Algorithm (SARAH)]],
*** a [[Stochastic Variance Reduced Gradient (SVRG)]].
* <B>See:</B> [[Stochastic Optimization]], [[Convex Optimization]], [[Learning Rate]], [[Gradient Descent]], [[Outer Product]], [[Hadamard Matrix Product]], [[Euclidean Norm]], [[Proximal Function]].

----
----