Leaky Rectified Linear Neuron

A Leaky Rectified Linear Neuron is an Rectified Linear Neuron in which the leakage coefficient is a non-zero small fraction, i.e. [math]\displaystyle{ 0\lt \alpha \lt \lt 1 }[/math].

• AKA: Leaky Rectified Linear Unit, Leaky ReLU.
• Example(s):
  • A Rectified Linear Neuron with [math]\displaystyle{ \alpha=0.01 }[/math]
  • a Randomized Leaky Rectified Linear Neuron,
  • …
• Counter-Example(s):
  • a Parametric Rectified Linear Neuron,
  • a S-shaped Rectified Linear Neuron,
  • an Adaptive Linear Neuron,
  • a Scaled Exponential Linear Neuron,
  • a Exponential Linear Neuron,
  • a Bent Identity Neuron.
  • a Hyperbolic Tangent Neuron,
  • a Sigmoid Neuron,
  • a Heaviside Step Neuron,
  • a Stochastic Binary Neuron,
  • a SoftPlus Neuron,
  • a Softmax Neuron.
• See: Artificial Neural Network, Perceptron, Linear Neuron.

---

---

References

2017

• (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
  • QUOTE: Leaky rectified linear unit (Leaky ReLU) — Leaky ReLUs allow a small, non-zero gradient when the unit is not active. 0.01 is the small non-zero gradient here

    [math]\displaystyle{ f(x) = \begin{cases} 0, & \mbox{for } 0.01x \lt 0 \\ x, & \mbox{for } x \geq 0 \end{cases} }[/math]

    Range:[math]\displaystyle{ (-\infty, +\infty) }[/math]

    

---