Softmax

[!def] Softmax
$$
\begin{align*}
Softmax &= \frac{e^x}{\sum_y e^y} \\ > &= \frac{e^x}{e^x + e^y}
\end{align*}
$$

• Range: $[0, 1]$

Derivation of Softmax

Derivative w.r.t. x

$$
\begin{align*}
f(x) &= \frac{e^x}{e^x + e^y} \\\frac{df(x)}{dx} &= \frac{(e^x + e^y) \frac{d}{dx} e^x - e^x \frac{d}{dx}(e^x + e^y)} {(e^x + e^y)^2} \\&= \frac{(e^x + e^y)e^x - e^x e^x}{(e^x + e^y)^2} \\&= \frac{e^x}{(e^x + e^y)} (\frac{(e^x + e^y)}{(e^x + e^y)} - \frac{e^x}{(e^x + e^y)}) \\&= f(x) (1 - f(x))
\end{align*}
$$

Derivative w.r.t y

$$
\begin{align*}
f(x) &= \frac{e^x}{e^x + e^y} \\\frac{df(x)}{dy} &= \frac{(e^x + e^y) \frac{d}{dy} e^x - e^x \frac{d}{dy}(e^x + e^y)} {(e^x + e^y)^2} \\&= \frac{(e^x + e^y) 0 - e^x e^y}{(e^x + e^y)^2} \\&= - \frac{e^x}{e^x + e^y} . \frac{e^y}{e^x+e^y} \\&= - f(x) . f(y)
\end{align*}
$$