MomentumNesterov

class brainpy.optim.MomentumNesterov(lr, train_vars=None, weight_decay=None, momentum=0.9, name=None)

Nesterov accelerated gradient optimizer [2].

\[\begin{split}\begin{align} \begin{split} v_t &= \gamma v_{t-1} + \eta \nabla_\theta J( \theta - \gamma v_{t-1} ) \\ \theta &= \theta - v_t \end{split} \end{align}\end{split}\]

Parameters:

lr (float, Scheduler) – learning rate.

References

[2]

Nesterov, Y. (1983). A method for unconstrained convex minimization problem with the rate of convergence o(1/k2). Doklady ANSSSR (translated as Soviet.Math.Docl.), vol. 269, pp. 543– 547.