brainpy.optim module#

Optimizers#

Optimizer(lr[, train_vars, name])

Base Optimizer Class.

SGD(lr[, train_vars, weight_decay, name])

Stochastic gradient descent optimizer.

Momentum(lr[, train_vars, momentum, ...])

Momentum optimizer.

MomentumNesterov(lr[, train_vars, ...])

Nesterov accelerated gradient optimizer [2]_.

Adagrad(lr[, train_vars, weight_decay, ...])

Optimizer that implements the Adagrad algorithm.

Adadelta([lr, train_vars, weight_decay, ...])

Optimizer that implements the Adadelta algorithm.

RMSProp(lr[, train_vars, weight_decay, ...])

Optimizer that implements the RMSprop algorithm.

Adam(lr[, train_vars, beta1, beta2, eps, ...])

Optimizer that implements the Adam algorithm.

LARS(lr[, train_vars, momentum, ...])

Layer-wise adaptive rate scaling (LARS) optimizer [1]_.

Adan([lr, train_vars, betas, eps, ...])

Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models [1]_.

AdamW(lr[, train_vars, beta1, beta2, eps, ...])

Adam with weight decay regularization [1]_.

Schedulers#

make_schedule(scalar_or_schedule)

Scheduler(lr[, last_epoch])

The learning rate scheduler.

Constant(lr[, last_epoch])

StepLR(lr, step_size[, gamma, last_epoch])

Decays the learning rate of each parameter group by gamma every step_size epochs.

MultiStepLR(lr, milestones[, gamma, last_epoch])

Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones.

CosineAnnealingLR(lr, T_max[, eta_min, ...])

Set the learning rate of each parameter group using a cosine annealing schedule, where \(\eta_{max}\) is set to the initial lr and \(T_{cur}\) is the number of epochs since the last restart in SGDR:

CosineAnnealingWarmRestarts(lr, ...[, ...])

Set the learning rate of each parameter group using a cosine annealing

ExponentialLR(lr, gamma[, last_epoch])

Decays the learning rate of each parameter group by gamma every epoch.

ExponentialDecay(lr, decay_steps, decay_rate)

InverseTimeDecay(lr, decay_steps, decay_rate)

PolynomialDecay(lr, decay_steps, final_lr[, ...])

PiecewiseConstant(boundaries, values[, ...])