Surrogate gradient

@Sichao He

In recent years, spiking neural networks (SNNs) show their promising advantages in energy efficiency, fault tolerance, and biological plausibility. However, SNNs are difficult to train using standard gradient descent methods because their activation functions are discontinuous and have zero gradients almost everywhere. The commonly used way is to replace the non-differentiable spiking function with the surrogate gradient function. A surrogate gradient function is a smooth function that approximates the derivative of the activation function and allows gradient-based learning algorithms to be applied to SNNs.

BrainPy provides multiple surrogate gradient functions with different properties of smoothness, boundedness, and biological plausibility. The full list is shown in Table below, and for the example of the surrogate gradient function please see the Figure Below.

In practice, users can use these surrogate gradient functions as parameters in neuron models. For example, in the leaky integrate-and-fire (LIF) neuron model brainpy.neurons.LIF, use can use:

model = brainpy.neurons.LIF (... , spike_fun=<surrogate function >)

Name	Implementation
Sigmoid function	brainpy.math.surrogate.sigmoid
Piecewise quadratic function	brainpy.math.surrogate.piecewise_quadratic
Piecewise exponential function	brainpy.math.surrogate.piecewise_exp
Soft sign function	brainpy.math.surrogate.soft_sign
Arctan function	brainpy.math.surrogate.arctan
Nonzero sign log function	brainpy.math.surrogate.nonzero_sign_log
Erf function	brainpy.math.surrogate.erf
Piecewise leaky relu function	brainpy.math.surrogate.piecewise_leaky_relu
Squarewave Fourier series	brainpy.math.surrogate.squarewave_fourier_series
S2NN surrogate spiking function	brainpy.math.surrogate.s2nn
q-PseudoSpike surrogate function	brainpy.math.surrogate.q_pseudo_spike
Leaky ReLU function	brainpy.math.surrogate.leaky_relu
Log-tailed ReLU function	brainpy.math.surrogate.log_tailed_relu
ReLU gradient function	brainpy.math.surrogate.relu_grad
Gaussian gradient function	brainpy.math.surrogate.gaussian_grad
Multi-Gaussian gradient function	brainpy.math.surrogate.multi_gaussian_grad
Inverse-square surrogate gradient	brainpy.math.surrogate.inv_square_grad
Slayer surrogate gradient function	brainpy.math.surrogate.slayer_grad