Overview

  • Tango 2 uses DPO to improve a text-to-audio diffusion model
  • problem: the base model drops sounds or puts them in the wrong order, and there is no answer to check against
  • so the preference pairs are made on purpose (Audio-Alpaca) in three ways: fewer denoising steps, swapped or dropped events, and reordered events
  • main lesson: a small set of made-up pairs can beat a model trained on far more audio

Tango 2 applies preference optimisation to text-to-audio generation. It shows that DPO works outside text, because DPO only needs a winner, a loser and a reference model.

Base model: Tango

  • Tango is a latent diffusion model over audio
  • It takes in a prompt and generates audio
  • the audio is turned into a mel spectrogram, and a VAE (audio encoder and audio decoder) maps it to and from a latent
  • the diffusion model has a UNet backbone and runs the forward process (adding noise) and the reverse process (removing noise) in the latent space
  • a frozen FLAN-T5 text encoder turns the prompt, for example "A dog is barking and growling, as a siren is blaring", into the condition
  • a HiFi-GAN vocoder turns the generated spectrogram into a waveform
  • Tango was trained on 63 times fewer samples than AudioLDM

Problem

The failure of Tango is not fluency. Its outputs:

  • drop acoustic events that are named in the prompt
  • get the temporal order of the events it keeps wrong

There is no verifiable reward and no gold answer. In maths or code, an output can be checked. In text-to-audio there is nothing to check against, so preference data must be manufactured.

Alignment with DPO

For the same prompt, one generation keeps every event in the correct order and the other does not. Tango (pretrained and fine-tuned) is aligned with a diffusion version of the DPO loss on the Audio-Alpaca preference dataset, which gives Tango 2 (pretrained, fine-tuned and aligned). Compared with Tango, Tango 2 improves the CLAP score by 5.5 and the human relevance rating (REL) by 7% on multi-event audio, and by 5.5 and 9% on audio where the temporal order matters.

Audio-Alpaca: manufacturing preference pairs

The prompts come from AudioCaps. A loser is made for each prompt in three ways:

  • strategy 1: one prompt, four generations with different numbers of denoising steps (or the same number of steps); the weakest output becomes the loser
  • strategy 2: GPT-4 rewrites the prompt so that an event is swapped or dropped, for example "A person yawns as people laugh" instead of "A person burps as people laugh"; audio generated from the rewrite is the loser
  • strategy 3: GPT-4 reorders the events in time, so the loser has the right sounds in the wrong order

Audio-Alpaca statistics

  • 15,025 preference pairs in total
  • every strategy gives a winner with a CLAP score near 0.645 and a loser near 0.452, so the average margin is 0.193
  • the margin is the training signal, not the absolute score, because the Bradley-Terry model only ever sees a difference

Results

Diffusion-DPO on about 15k synthetic pairs beats a model trained on far more audio. Tango 2 also keeps a higher CLAP score as the number of events or concepts in the prompt increases. Removing any strategy from the data makes the results worse.

What carried over from language

  • carried over: the Bradley-Terry pairwise formulation, the reference-model anchor, and the fact that a small preference set beats a large demonstration set
  • did not carry over: there are no human annotators and no verifiable reward, so the preference pairs had to be synthesised from the model's own failure modes
  • result: better event coverage, better temporal ordering and better prompt adherence, on both automatic and human evaluation
  • open question: Audio-Alpaca is built once and then frozen, so what happens when the model outgrows the dataset that trained it

Note

The answer to the open question is to regenerate the preference data during training, which is the idea of TangoFlux.