SpecAugment is a data augmentation that works directly on a spectrogram. It hides random bands of frequency and random stretches of time, so a model cannot depend on one narrow part of the input. It was introduced for speech recognition and also works for other signals shown as spectrograms. Spectrogram inputs are covered in Audio Representations.
Operations
- frequency masking: set consecutive frequency bins to zero (or the mean), with drawn from 0 to a maximum
- time masking: set consecutive time frames to zero, with from 0 to a maximum
- time warping (in the original paper): stretch part of the time axis slightly, often left out because it helps less
Why It Works
- forces the model to use many cues instead of one
- makes the model robust to missing or corrupted parts of a signal, for example a narrow interferer or a short dropout
- it is cheap, because it runs on the spectrogram tensor on the GPU
In PyTorch
import torchaudio.transforms as T
augment = torch.nn.Sequential(
T.FrequencyMasking(freq_mask_param=102), # up to 10% of 1024 bins
T.TimeMasking(time_mask_param=20),
)
x = augment(x) # training onlyRelated Signal Augmentations
- adding white noise at a random signal-to-noise ratio, so the model learns weak signals
- random gain and phase errors between I and Q for radio signals
- random frequency shifts