brainpy.nn.nodes.ANN.LayerNorm
brainpy.nn.nodes.ANN.LayerNorm#
- class brainpy.nn.nodes.ANN.LayerNorm(epsilon=1e-05, use_bias=True, use_scale=True, beta_init=ZeroInit, gamma_init=OneInit(value=1.0), axis=None, **kwargs)[source]#
Layer normalization (https://arxiv.org/abs/1607.06450).
This layer normalizes data on each example, independently of the batch. More specifically, it normalizes data of shape (b, d1, d2, …, c) on the axes of the data dimensions and the channel (d1, d2, …, c). Different from batch normalization, gamma and beta are assigned to each position (elementwise operation) instead of the whole channel. If users want to assign a single gamma and beta to a whole example/whole channel, please use GroupNorm/ InstanceNorm.
- Parameters
epsilon (float) – a value added to the denominator for numerical stability. Default: 1e-5
use_bias (bool) – whether to translate data in refactoring. Default: True
use_scale (bool) – whether to scale data in refactoring. Default: True
beta_init (brainpy.init.Initializer) – an initializer generating the original translation matrix
gamma_init (brainpy.init.Initializer) – an initializer generating the original scaling matrix
axis (int, tuple, list) – axes where the data will be normalized. The batch axis should be excluded.
- __init__(epsilon=1e-05, use_bias=True, use_scale=True, beta_init=ZeroInit, gamma_init=OneInit(value=1.0), axis=None, **kwargs)[source]#
Methods
__init__
([epsilon, use_bias, use_scale, ...])copy
([name, shallow])Returns a copy of the Node.
default_axis
()feedback
(ff_output, **shared_kwargs)The feedback computation function of a node.
forward
(ff, **shared_kwargs)The feedforward computation function of a node.
init_fb_conn
()Initialize the feedback connections.
init_fb_output
([num_batch])Set the initial node feedback state.
init_ff_conn
()Initialize the feedforward connections.
init_state
([num_batch])Set the initial node state.
initialize
([num_batch])Initialize the node.
load_states
(filename[, verbose])Load the model states.
nodes
([method, level, include_self])Collect all children nodes.
offline_fit
(targets, ffs[, fbs])Offline training interface.
online_fit
(target, ff[, fb])Online training fitting interface.
online_init
()Online training initialization interface.
register_implicit_nodes
(nodes)register_implicit_vars
(variables)save_states
(filename[, variables])Save the model states.
set_fb_output
(state)Safely set the feedback state of the node.
set_feedback_shapes
(fb_shapes)set_feedforward_shapes
(feedforward_shapes)set_output_shape
(shape)set_state
(state)Safely set the state of the node.
train_vars
([method, level, include_self])The shortcut for retrieving all trainable variables.
unique_name
([name, type_])Get the unique name for this object.
vars
([method, level, include_self])Collect all variables in this node and the children nodes.
Attributes
data_pass
Offline fitting method.
fb_output
feedback_shapes
Output data size.
feedforward_shapes
Input data size.
is_feedback_input_supported
is_feedback_supported
is_initialized
- rtype
name
output_shape
Output data size.
state
Node current internal state.
trainable
Returns if the Node can be trained.