Perplexity

• Perplexity is an Intrinsic Evaluation
• Using Extrinsic Evaluation or downstream tasks is slow and hard
• So people use Perplexity to compare different models and with other research
• Perplexity tells us how good the model is generating sentences
• Less Perplexity is better

[!def] Perplexity from Likelihood
$$
\begin{align*}
PPL &= P(w_1 w_2 ... w_N)^{-\frac{1}{N}} \\ > &= \sqrt[N]{\frac{1}{P(w_1 w_2 ... w_N)}} \\ > &= \sqrt[N]{\frac{1}{\prod_i P(w_i|w_1 w_2 ... w_{i-1})}} \\ > \end{align*}
$$

[!def] Perplexity from Cross Entropy
$$
PPL = e^{CE}
$$