brainpy.optim.RMSProp#
- class brainpy.optim.RMSProp(lr, train_vars=None, weight_decay=None, epsilon=1e-06, rho=0.9, name=None)[source]#
Optimizer that implements the RMSprop algorithm.
RMSprop [5] and Adadelta have both been developed independently around the same time stemming from the need to resolve Adagrad’s radically diminishing learning rates.
The gist of RMSprop is to:
Maintain a moving (discounted) average of the square of gradients
Divide the gradient by the root of this average
\[\begin{split}\begin{split}c_t &= \rho c_{t-1} + (1-\rho)*g^2\\ p_t &= \frac{\eta}{\sqrt{c_t + \epsilon}} * g \end{split}\end{split}\]The centered version additionally maintains a moving average of the gradients, and uses that average to estimate the variance.
References
Methods
__init__
(lr[, train_vars, weight_decay, ...])check_grads
(grads)cpu
()Move all variable into the CPU device.
cuda
()Move all variables into the GPU device.
load_state_dict
(state_dict[, warn, compatible])Copy parameters and buffers from
state_dict
into this module and its descendants.load_states
(filename[, verbose])Load the model states.
nodes
([method, level, include_self])Collect all children nodes.
register_implicit_nodes
(*nodes[, node_cls])register_implicit_vars
(*variables[, var_cls])register_train_vars
([train_vars])register_vars
([train_vars])save_states
(filename[, variables])Save the model states.
state_dict
()Returns a dictionary containing a whole state of the module.
to
(device)Moves all variables into the given device.
tpu
()Move all variables into the TPU device.
train_vars
([method, level, include_self])The shortcut for retrieving all trainable variables.
tree_flatten
()Flattens the object as a PyTree.
tree_unflatten
(aux, dynamic_values)Unflatten the data to construct an object of this class.
unique_name
([name, type_])Get the unique name for this object.
update
(grads)vars
([method, level, include_self, ...])Collect all variables in this node and the children nodes.
Attributes
name
Name of the model.