Whisper is a speech recognition model from OpenAI (Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, 2022). It takes audio and outputs text. large-v2 is the largest model of the original release. It has the same architecture as the first large model, but it was trained for 2.5 times more epochs with extra regularisation, and it gives lower error rates.

Weak supervision

Whisper is trained on 680,000 hours of audio paired with transcripts collected from the internet. The transcripts were filtered but not checked by hand, so the labels are "weak".

  • the large amount of varied data makes the model robust to accents, background noise, and recording quality
  • the model works zero-shot on new datasets, with no fine-tuning
  • the same data has many languages, so one model handles transcription in about 99 languages and translation into English

Model sizes

ModelLayersWidthHeadsParameters
tiny4384639M
base6512874M
small1276812244M
medium24102416769M
large, large-v2321280201550M

The large models need about 10 GB of GPU memory and are the slowest of the set.


Architecture

Whisper is an encoder-decoder Transformer, which is a Seq2Seq model where the source sequence is audio and the target sequence is text.

Audio front end

  1. resample the audio to 16 kHz
  2. pad or cut the audio to exactly 30 seconds, which is 480,000 samples
  3. compute an 80-channel log-Mel spectrogram with a 25 ms window and a 10 ms step
  4. scale the values to about the range
  5. pass the spectrogram through 2 convolution layers (filter width 3, GELU activation), where the second layer has stride 2
  6. add sinusoidal position encodings and pass the result to the encoder blocks

The frame count follows from the step size and the stride:

So each encoder output frame covers about 20 ms of audio.

Decoder

  • the decoder uses learned position embeddings
  • it predicts one text token at a time, conditioned on the tokens so far
  • it uses cross-attention on the 1500 encoder frames, so each output word can look at the part of the audio where it was spoken
  • the tokenizer is byte-level BPE, the same type as GPT-2, refitted for many languages

Multitask token format

One model does several tasks. The task is chosen by special tokens at the start of the decoder input:

<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|> put the red cube in the blue bowl <|endoftext|>
TokenPurpose
<|startoftranscript|>starts every output
language token, for example <|en|>the spoken language, which the model can also predict by itself
<|transcribe|> or <|translate|>write the speech in its own language, or translate it into English
<|notimestamps|>text only, with no time tokens
timestamp tokensstart and end time of each segment, in steps of 20 ms
<|nospeech|>the 30 second window has no speech

Forcing the language and task tokens is a simple way to stop the model from detecting the wrong language on a short clip.


Long audio

The model only sees 30 seconds at a time. For longer audio, the transcribe function moves the window forward using the timestamps that the model predicts. It also uses these checks to catch bad outputs:

CheckDefaultWhat it catches
compression ratio threshold2.4repeated text loops, which compress very well
log probability thresholdlow-confidence output
no speech threshold0.6windows that are silence or noise
temperature fallbackdecode again with more randomness when a check fails

Whisper can hallucinate. On silence or noise it may repeat a phrase or write text that was never spoken. For a robot, a false command is worse than no command, so check no_speech_prob and avg_logprob before you pass the text to the planner.

Evaluation

Speech recognition is scored with the word error rate:

Where is the number of substituted words, is deleted words, is inserted words, and is the number of words in the reference transcript. A lower WER is better.


Code

openai-whisper

import whisper
 
model = whisper.load_model("large-v2")  # downloads about 3 GB on first use
 
result = model.transcribe(
    "command.wav",
    language="en",
    task="transcribe",
    fp16=True,  # set to False on CPU
)
 
print(result["text"])
 
for segment in result["segments"]:
    print(f'{segment["start"]:.2f}s to {segment["end"]:.2f}s: {segment["text"]}')
    print(f'  no_speech_prob = {segment["no_speech_prob"]:.2f}')
  • transcribe does the resampling, padding, and spectrogram steps for you, but it needs ffmpeg installed to read the audio file
  • result["segments"] holds each segment with its timestamps and confidence values
  • language="en" forces the language token, so no language detection step runs

Hugging Face transformers

import torch
from transformers import pipeline
 
asr = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-large-v2",
    torch_dtype=torch.float16,
    device="cuda:0",
)
 
output = asr(
    "command.wav",
    chunk_length_s=30,
    return_timestamps=True,
    generate_kwargs={"language": "english", "task": "transcribe"},
)
 
print(output["text"])
ArgumentMeaning
chunk_length_ssplits long audio into 30 second chunks, matching the model window
return_timestampsreturns the start and end time of each chunk of text
generate_kwargsforces the language and task tokens in the decoder prompt

large-v2 vs large-v3

large-v3 uses 128 Mel channels instead of 80, so its input features are not the same as large-v2. If you change model, use the matching processor and do not reuse cached large-v2 spectrograms.