ReLU

• ReLU = Rectified Linear Unit
• It solves the Vanishing Gradient problem
• But introduces Dying ReLU problem
• Most popular function for CNN or even #deep-learning models

[!def] ReLU Formula
$$
f(x) = max(0, x)
$$

Derivative of ReLU

$$
f'(x) = 0 \text{ if x <= 0}
$$
$$
f'(x) = 1 \text{ if x > 0}
$$