Scaling Audio-Text Retrieval with Multimodal Large Language Models

Jilan Xu, Carl Thome, Danijela Horak, Weidi Xie, Andrew Zisserman

Visual Geometry Group, University of Oxford ยท Epidemic Sound

Technical Report

Paper arXiv Code HF Model

TL;DR

We propose AuroLA, a contrastive language-audio pre-training framework that repurposes a Multimodal LLM as a unified backbone for retrieval. The system builds scalable multi-granular audio captions, uses a Hybrid-NCE objective with hard-negative reweighting, and applies a bidirectional re-ranker to refine results. Experiments show consistent gains over prior audio-text retrieval SoTA, i.e., Perception Encoder Audio-Visual (PE-AV), with approximately 1% of their training data.

Visualisation

Text query: Footsteps shuffling on snow alongside a camera muffling while wind blows into a microphone
Text query: A vehicle engine revving several times with a series of compressed air releasing and plastic pops
Text query: Railroad tracks clickety-clack as a train horn blasts
Text query: Music plays continuously along with swooshing followed by a man speaking quickly
Text query: Frogs croaking and a humming with insects vocalizing

Model Architecture

AuroLA model architecture

Overall architecture of AuroLA and AuroLA-re-rank

Left: Re-purposing multimodal large language model for fast retrieval by explicit one-word limitation.

Right: Re-purposing multimodal large language model for precise re-ranking.

Results

Comparison with SOTA dual encoders

AudioCaps text-to-audio comparison Clotho text-to-audio comparison

Scaling properties of AuroLA

AudioCaps data scaling results Clotho data scaling results