InstructEval is a holistic evaluation suite for instruction tuned language models. It exists because a single leaderboard number hides what instruction tuning did and did not buy.
Three axes measured
- problem solving
- writing ability
- alignment to human values
Inference speed is not one of them.
Headline findings
- instruction-data quality matters more than quantity
- a model's score is attributed to three causes: the pretraining foundation, the instruction data and the training method
- open source instruction tuned models are fluent, and fluency is not a good proxy for capability, because they lag on problem solving and alignment
The training method it evaluates is Instruction Tuning.