Overview

  • sound is first turned into a picture-like spectrogram, then encoded much like an image
  • path: waveform, then STFT, then log-Mel spectrogram, then patches
  • three speech models: wav2vec 2.0 (contrastive), HuBERT (masked prediction) and Whisper (weak supervision)
  • big idea: audio uses the same three tricks as vision

An audio representation is the vector form of a sound clip. Raw audio is long and redundant, so it is first converted into an image-like form and then encoded with the same ideas used for vision.

From waveform to spectrogram

  • a waveform shows amplitude over time
  • speech is usually sampled at 16 kHz, which is 16,000 numbers per second, so the raw signal is long and redundant
  • the short-time Fourier transform (STFT) cuts the sound into short windows and finds the frequencies in each window, which turns the sound into a time-frequency image called a spectrogram (energy per frequency over time)
  • a log-Mel spectrogram rescales the frequency axis to match how humans perceive pitch
  • the result can be cut into patches and fed to a Transformer, just like an image (this is the idea of the Audio Spectrogram Transformer)

Learning speech representations

Three well-known models:

  • wav2vec 2.0 (2020): a CNN encodes the raw waveform, spans of time steps are masked, and the model is trained contrastively to pick the true (quantised) latent for a masked step from among distractors
  • HuBERT (2021): audio features are first clustered offline with k-means; spans of time steps are masked, and the model does masked prediction of the hidden cluster IDs, like BERT predicts hidden words
  • Whisper (2022): a Transformer encoder-decoder trained with weak supervision on 680,000 hours of web audio with transcripts; it does multilingual speech recognition and translation

Note

Audio uses the same three ideas as vision: contrast, mask and predict, and weak labels. A detailed note on one version of Whisper is Whisper large-v2.