Softplus

• Smoother version of ReLU
• Negative values still get 0
• But for positive ones, its smoother and unbounded
  • Improve training stability
  • But computation is expensive due to complex gradient

[!def] Softplus Formula
$$
f(x) = log(1 + \exp(x))
$$