Tanh

• Tanh = Hyperbolic Tangent Function
• Range: $[-1, 1]$

[!def] TanH Formula
$$
f(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}
$$

• Compared to Sigmoid Function, the gradient near 0 is notably larger, leading to more substantial update
• But it also saturates, which might cause Vanishing Gradient

TODO:

1. Add derivative