LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining
Abstract
Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces LAION-Audio-XL, a 365M audio-text dataset, and CLAP2, a family of contrastive audio-language models, demonstrating that data scaling is more effective than model scaling for contrastive pretraining within the tested compute budget. The comprehensive data curation pipeline and rigorous scaling analysis provide valuable insights for the field, while the release of the dataset and models facilitates future research.
Methodology
The paper presents a robust and well-documented methodology for constructing a large-scale audio-text dataset (LAION-Audio-XL) and training contrastive models (CLAP2). The data pipeline is extensive, involving multi-stage curation including exact and near-duplicate removal (using Olaf fingerprints and SemDeDup), signal quality checks, and a sophisticated taxonomy-based selection process to ensure diversity. The use of a 4,416-node taxonomy for coverage-aware selection is a strong methodological contribution. The model architecture utilizes a NaFlex audio encoder and a ModernBERT-based text encoder, trained from scratch to avoid confounding effects from pretraining. The scaling study methodology is rigorous, employing warmup-stable-decay learning rate schedules and fitting power laws to downstream error rather than just validation loss, which provides more practical insights for compute allocation.
Results & evaluation
The experimental evaluation is comprehensive, covering five model sizes (27M to 340M parameters) and eleven data scales (1M to 1024M samples). The results on MAEB-30, HEAR, and MSEB benchmarks show that the models outperform prior audio-text models of comparable size. The scaling analysis reveals that data scaling is more effective than model scaling for contrastive audio-language pretraining within the tested range, with optimal compute allocation favoring more samples over larger models. The ablation studies on audio length (30s vs 120s) and caption length provide useful practical insights. The comparison with larger models (1B+ parameters) demonstrates the efficiency of the proposed approach.
Reproducibility
The paper is highly reproducible. The authors release the dataset, captions, model checkpoints, and code. The data pipeline is described in detail in the appendix, including specific thresholds for deduplication and selection. The training hyperparameters are provided, and the use of standard libraries (timm, faiss, etc.) facilitates implementation. The release of the 365M sample dataset is a significant contribution to the community, enabling further research on scaling laws and data quality.
Limitations
The primary limitation is the reliance on web-scraped data, which may contain biases and noise despite the extensive curation pipeline. The models are evaluated primarily on classification and retrieval tasks, with limited evaluation on generative tasks or other downstream applications. The scaling study is limited to models up to 340M parameters and data up to 1B samples, so the behavior at larger scales remains unknown. The use of model-generated captions for a large portion of the dataset may introduce captioning errors that affect model performance.
Broader impact
The release of LAION-Audio-XL and CLAP2 models will significantly impact the audio-language research community by providing a large, high-quality dataset and strong baseline models. The scaling insights will guide future research in compute allocation for audio pretraining. The dataset's diversity and size enable research on long-tail sound events and multilingual audio understanding. The open-source nature of the work promotes transparency and reproducibility in audio-language learning. The paper introduces LAION-Audio-XL, a 365M audio-text dataset, and CLAP2, a family of contrastive audio-language models, demonstrating that data scaling is more effective than model scaling for contrastive pretraining within the tested compute budget. The comprehensive data curation pipeline and rigorous scaling analysis provide valuable insights for the field, while the release of the dataset and models facilitates future research.
Author of this paper? Send a correction.