After a model produces an output, there is a need to evaluate its quality quantitatively rather than by impression. Evaluation metrics give a number that can be compared across runs.
Evaluation metrics assist us in 3 tasks:
- Objective Comparisons: a common framework for assessing and comparing the performance of different models
- Model Selection: informed choices when picking the best performing model among several candidates
- Hyperparameter Tuning: assessing the impact of hyperparameter choices, so the best set can be selected
Types of Models
Text models are generally split into 2 types, and each type requires different kinds of evaluation metrices.
| Type | What it does | Examples |
|---|---|---|
| Predictive | trained to make predictions or classifications from input data | classification models, regression models |
| Generative | trained to create new data, usually text | AI chatbots, translators, text summarisers |
Metrics for predictive models:
Metrics for generative models:
- Bilingual Evaluation Understudy (BLEU)
- Recall-Oriented Understudy for Gisting Evaluation (ROUGE)
- Metric for Evaluation of Translation with Explicit Ordering (METEOR)
Precision, recall and F1 can each be reported in 2 ways when there is more than 1 class, which is covered in Micro and Macro Metrics.
The outputs being measured here come from the models in Text Classification.