Randomized Leaky Rectified Linear Neuron

A Randomized Leaky Rectified Linear Neuron is a Leaky Rectified Linear Neuron in which the leakage coefficient is a random variable.

• AKA: Randomized Leaky Rectified Linear Unit, RReLU.
• Example(s):
  • tfa.activations.rrelu a TensorFlow implementation,
  • torch.nn.RReLu a Pytorch implmenation,
  • …
• Counter-Example(s):
  • a Parametric Rectified Linear Neuron,
  • a S-shaped Rectified Linear Neuron,
  • an Adaptive Linear Neuron,
  • a Scaled Exponential Linear Neuron,
  • a Exponential Linear Neuron,
  • a Bent Identity Neuron.
  • a Hyperbolic Tangent Neuron,
  • a Sigmoid Neuron,
  • a Heaviside Step Neuron,
  • a Stochastic Binary Neuron,
  • a SoftPlus Neuron,
  • a Softmax Neuron.
• See: Artificial Neural Network, Perceptron, Linear Neuron.

---

---

References

2017

• (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
  • QUOTE: Randomized Leaky Rectified Linear Unit(RReLU)

    Range:[math]\displaystyle{ (-\infty, +\infty) }[/math]

    [math]\displaystyle{ f(\alpha, x) = \begin{cases} \alpha x, & \mbox{for } x \lt 0 \\ x, & \mbox{for } x \geq 0 \end{cases} }[/math]

    

---