Whisper is a speech recognition model from OpenAI (Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, 2022). It takes audio and outputs text. large-v2 is the largest model of the original release. It has the same architecture as the first large model, but it was trained for 2.5 times more epochs with extra regularisation, and it gives lower error rates.
Weak supervision
Whisper is trained on 680,000 hours of audio paired with transcripts collected from the internet. The transcripts were filtered but not checked by hand, so the labels are "weak".
- the large amount of varied data makes the model robust to accents, background noise, and recording quality
- the model works zero-shot on new datasets, with no fine-tuning
- the same data has many languages, so one model handles transcription in about 99 languages and translation into English
Model sizes
| Model | Layers | Width | Heads | Parameters |
|---|---|---|---|---|
| tiny | 4 | 384 | 6 | 39M |
| base | 6 | 512 | 8 | 74M |
| small | 12 | 768 | 12 | 244M |
| medium | 24 | 1024 | 16 | 769M |
| large, large-v2 | 32 | 1280 | 20 | 1550M |
The large models need about 10 GB of GPU memory and are the slowest of the set.
Architecture
Whisper is an encoder-decoder Transformer, which is a Seq2Seq model where the source sequence is audio and the target sequence is text.
Audio front end
- resample the audio to 16 kHz
- pad or cut the audio to exactly 30 seconds, which is 480,000 samples
- compute an 80-channel log-Mel spectrogram with a 25 ms window and a 10 ms step
- scale the values to about the range
- pass the spectrogram through 2 convolution layers (filter width 3, GELU activation), where the second layer has stride 2
- add sinusoidal position encodings and pass the result to the encoder blocks
The frame count follows from the step size and the stride:
So each encoder output frame covers about 20 ms of audio.
Decoder
- the decoder uses learned position embeddings
- it predicts one text token at a time, conditioned on the tokens so far
- it uses cross-attention on the 1500 encoder frames, so each output word can look at the part of the audio where it was spoken
- the tokenizer is byte-level BPE, the same type as GPT-2, refitted for many languages
Multitask token format
One model does several tasks. The task is chosen by special tokens at the start of the decoder input:
<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|> put the red cube in the blue bowl <|endoftext|>
| Token | Purpose |
|---|---|
<|startoftranscript|> | starts every output |
language token, for example <|en|> | the spoken language, which the model can also predict by itself |
<|transcribe|> or <|translate|> | write the speech in its own language, or translate it into English |
<|notimestamps|> | text only, with no time tokens |
| timestamp tokens | start and end time of each segment, in steps of 20 ms |
<|nospeech|> | the 30 second window has no speech |
Forcing the language and task tokens is a simple way to stop the model from detecting the wrong language on a short clip.
Long audio
The model only sees 30 seconds at a time. For longer audio, the transcribe function moves the window forward using the timestamps that the model predicts. It also uses these checks to catch bad outputs:
| Check | Default | What it catches |
|---|---|---|
| compression ratio threshold | 2.4 | repeated text loops, which compress very well |
| log probability threshold | low-confidence output | |
| no speech threshold | 0.6 | windows that are silence or noise |
| temperature fallback | decode again with more randomness when a check fails |
Whisper can hallucinate. On silence or noise it may repeat a phrase or write text that was never spoken. For a robot, a false command is worse than no command, so check
no_speech_probandavg_logprobbefore you pass the text to the planner.
Evaluation
Speech recognition is scored with the word error rate:
Where is the number of substituted words, is deleted words, is inserted words, and is the number of words in the reference transcript. A lower WER is better.
Code
openai-whisper
import whisper
model = whisper.load_model("large-v2") # downloads about 3 GB on first use
result = model.transcribe(
"command.wav",
language="en",
task="transcribe",
fp16=True, # set to False on CPU
)
print(result["text"])
for segment in result["segments"]:
print(f'{segment["start"]:.2f}s to {segment["end"]:.2f}s: {segment["text"]}')
print(f' no_speech_prob = {segment["no_speech_prob"]:.2f}')transcribedoes the resampling, padding, and spectrogram steps for you, but it needsffmpeginstalled to read the audio fileresult["segments"]holds each segment with its timestamps and confidence valueslanguage="en"forces the language token, so no language detection step runs
Hugging Face transformers
import torch
from transformers import pipeline
asr = pipeline(
"automatic-speech-recognition",
model="openai/whisper-large-v2",
torch_dtype=torch.float16,
device="cuda:0",
)
output = asr(
"command.wav",
chunk_length_s=30,
return_timestamps=True,
generate_kwargs={"language": "english", "task": "transcribe"},
)
print(output["text"])| Argument | Meaning |
|---|---|
chunk_length_s | splits long audio into 30 second chunks, matching the model window |
return_timestamps | returns the start and end time of each chunk of text |
generate_kwargs | forces the language and task tokens in the decoder prompt |
large-v2 vs large-v3
large-v3 uses 128 Mel channels instead of 80, so its input features are not the same as large-v2. If you change model, use the matching processor and do not reuse cached large-v2 spectrograms.