Hyperbolic Tangent Activation Function

A Hyperbolic Tangent Activation Function is a Neuron Activation Function based on an Hyperbolic Tangent Function.

• AKA: TanH Activation Function.
• Context:
  • It can (typically) be used in the activation of Hyperbolic Tangent Neurons.
• Example(s):
  • torch.nn.Tanh [1]
• Counter-Example(s):
  • Hyperbolic Tangent-based Activation Functions:
    • a HardTanh Activation Function,
    • a Tanhshrink Activation Function,
  • a Rectified Linear Activation Function,
  • a Heaviside Step Activation Function,
  • a Ramp Activation Function,
  • a Logistic Sigmoid Activation Function,
  • a Gaussian Activation Function.
• See: Artificial Neural Network, Artificial Neuron, Neural Network Topology, Neural Network Layer, Neural Network Learning Rate.

---

---

References

2018a

• (Pytorch, 2018) ⇒ http://pytorch.org/docs/master/nn.html#tanh Retrieved:2018-2-10
  • QUOTE: class torch.nn.Tanh source

    Applies element-wise, [math]\displaystyle{ f(x)=\dfrac{\exp(x)−\exp(−x)}{\exp(x)+\exp(−x)} }[/math]

    Shape:

    • Input: (N,∗) where * means, any number of additional dimensions
    • Output: (N,∗), same shape as the input

Examples:

>>> m = nn.Tanh()
>>> input = autograd.Variable(torch.randn(2))
>>> print(input)
>>> print(m(input))

2018b

• (Santos, 2018) ⇒ Santos (2018) "Activation Functions". In: Neural Networks - Artificial Inteligence Retrieved: 2018-01-28.
  • QUOTE: After the neuron do the dot product between it's inputs and weights, it also apply a non-linearity on this result. This non-linear function is called Activation Function.

    On the past the popular choice for activation functions were the sigmoid and tanh. Recently it was observed the ReLU layers has better response for deep neural networks, due to a problem called vanishing gradient. So you can consider using only ReLU neurons.

    sigmoid: [math]\displaystyle{ \sigma(x)=\dfrac{1}{1+e^{−x}} }[/math]

    tanh:[math]\displaystyle{ \sigma(x)=\dfrac{e^x−e^x}{e^x+e^x} }[/math]

     ReLU:[math]\displaystyle{ \sigma(x)=max(0,x) }[/math]

    

2018c

• (CS231n, 2018) ⇒ Commonly used activation functions. In: CS231n Convolutional Neural Networks for Visual Recognition Retrieved: 2018-01-28.
  • QUOTE: Every activation function (or non-linearity) takes a single number and performs a certain fixed mathematical operation on it. There are several activation functions you may encounter in practice:
    • Tanh. The tanh non-linearity is shown on the image above on the right. It squashes a real-valued number to the range [-1, 1]. Like the sigmoid neuron, its activations saturate, but unlike the sigmoid neuron its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity. Also note that the tanh neuron is simply a scaled sigmoid neuron, in particular the following holds: [math]\displaystyle{ tanh(x)=2\sigma(2x)−1 }[/math].

2017

• (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
  • QUOTE: Hyperbolic tangent (TanH)  —  It looks like a scaled sigmoid function. Data is centered around zero, so the derivatives will be higher. Tanh quickly converges than sigmoid and logistic activation functions.

    [math]\displaystyle{ f(x)=\tanh(x)=\dfrac{2}{1+e^{-2x}} -2 }[/math]

    Range: [math]\displaystyle{ (-1, 1) }[/math]

    Examples: [math]\displaystyle{ \tanh(2) = 0.9640,\; \tanh(-0.567) = -0.5131, \; \tanh(0) = 0 }[/math]

    

2005

• (Golda,2005) ⇒ Adam Golda (2005). "Introduction to neural networks"
  • QUOTE: Functions that more accurate describe the non-linear characteristic of the biological neuron activation function are:
    • (...) hyperbolic tangent function:[math]\displaystyle{ y=tgh\left(\dfrac{\alpha\varphi}{2}\right)=\dfrac{1 - \exp({-\alpha\varphi})}{1+\exp({-\alpha\varphi})} }[/math] where [math]\displaystyle{ \alpha }[/math] is a parameter.

      The next picture presents the graphs of particular activation functions:

a. linear function,

b. threshold function,

c. sigmoid function.

---