Huber Loss

• Hybrid of Mean Squared Error (MSE) and Mean Absolute Error (MAE)
  • Squared error for small differences
  • Absolute error for large differences
• It has a hyperparameter $\delta$

[!def] Huber Loss Formula
$$
\begin{equation}
L =
\begin{cases}
\frac{1}{2} (y - \hat{y})^2, & \text{if } |y-\hat{y}| \leq \delta \\ > \delta |y - \hat{y}| - \frac{1}{2} \delta^2, & \text{otherwise}
\end{cases}
\end{equation}
$$

Pros

1. Differentiable at 0
2. good at Handling Outliers
3. The hyperparameter $\delta$ can be tuned to maximize model accuracy

Cons

1. Additional conditions make it computationally expensive
2. Differentiable only once