Ctrl+K

LARS

Contents

LARS#

class brainpy.optim.LARS(lr, train_vars=None, momentum=0.9, weight_decay=0.0001, tc=0.001, eps=1e-05, name=None)[source]#

Layer-wise adaptive rate scaling (LARS) optimizer [1].

Layer-wise Adaptive Rate Scaling, or LARS, is a large batch optimization technique. There are two notable differences between LARS and other adaptive algorithms such as Adam or RMSProp: first, LARS uses a separate learning rate for each layer and not for each weight. And second, the magnitude of the update is controlled with respect to the weight norm for better control of training speed.

\[\begin{split}m_{t} = \beta_{1}m_{t-1} + \left(1-\beta_{1}\right)\left(g_{t} + \lambda{x_{t}}\right) \\ x_{t+1}^{\left(i\right)} = x_{t}^{\left(i\right)} - \eta_{t}\frac{\phi\left(|| x_{t}^{\left(i\right)} ||\right)}{|| m_{t}^{\left(i\right)} || }m_{t}^{\left(i\right)}\end{split}\]
Parameters:

  • lr (float, Scheduler) – learning rate.

  • momentum (float) – coefficient used for the moving average of the gradient.

  • weight_decay (float) – weight decay coefficient.

  • tc (float) – trust coefficient eta ( < 1) for trust ratio computation.

  • eps (float) – epsilon used for trust ratio computation.

References

Contents