Emilia2: A Large-Scale Dataset for Controllable Long-Form Speech Generation
Abstract
Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per recording. We introduce Emilia2, a 229K-hour multilingual dataset for controllable long-form speech generation. From in-the-wild data, Emilia2 constructs short utterances, single-speaker long-form speech, and multi-speaker dialogues, preserving transcripts, speaker identities, pauses, and acoustic transitions. Examples longer than five minutes account for 50.0% of single-speaker long-form hours and 66.8% of dialogue hours. To support expressive control, Emilia2 provides natural language instructions for 4.77M examples and nonverbal-vocalization tags for 933K examples. We also introduce Emilia2-testset, 23 hours of bilingual recordings of up to 30 minutes and four speakers, each speaker paired with reference audio. We train autoregressive (AR) and non-autoregressive (NAR) models on Emilia2. Our 0.6B-backbone NAR model achieves the lowest Chinese CER and English WER on TTSD-Eval (5.27%/9.93%) among compared systems, while our AR model is on par with Qwen3-TTS-12Hz-VD on InstructTTSEval (zh/en: 76.2/78.3 vs. 77.1/77.9). These results validate Emilia2 as a comprehensive data foundation for long-form speech generation and minute-scale expressive control. Demo: https://charlesnii.github.io/Emilia2-demo/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Emilia2, a 229K-hour multilingual dataset and benchmark for controllable long-form speech generation, demonstrating that models trained on this data achieve state-of-the-art performance in both long-form consistency and expressive control.
Methodology
The paper proposes a comprehensive data construction pipeline, Emilia2, that transforms in-the-wild audio into a structured dataset for long-form speech generation. The methodology is robust, utilizing a multi-stage process involving audio standardization, speaker diarization (DiariZen), VAD (FireRedVAD), and quality filtering (DNSMOS). A key methodological strength is the preservation of temporal continuity and acoustic transitions in long-form examples, rather than concatenating trimmed segments, which is crucial for modeling natural pauses and turn-taking. The integration of LLM-based annotation (Qwen3-Omni) for style instructions and specialized ASR for nonverbal vocalizations (NV) adds significant value for controllable generation. The training of both Autoregressive (AR) and Non-Autoregressive (NAR) models on this data validates the dataset's versatility.
Results & evaluation
The experimental evaluation is extensive, covering long-form generation (TTSD-Eval, Long-TTS-Eval, Emilia2-testset) and short-form controllable generation (InstructTTSEval, NV-Bench). The introduction of Emilia2-testset, featuring up to 30-minute recordings with up to 4 speakers and per-speaker reference audio, addresses a critical gap in existing benchmarks. The results show that the 0.6B backbone models trained on Emilia2 outperform larger baselines (e.g., 3B Higgs Audio v2) in objective metrics like WER/CER and speaker similarity. Subjective evaluations (MOS) further support the naturalness and control capabilities. The comparison against state-of-the-art systems like Qwen3-TTS and MOSS-TTSD is relevant and competitive.
Reproducibility
The paper provides detailed descriptions of the pipeline components (specific models used for diarization, ASR, etc.) and the training curriculum. However, the source data is described as "in-the-wild" from various internet sources, which may raise questions about licensing and exact reproducibility of the raw data collection. The release of the dataset and testset is a major positive for reproducibility. The code for the models is not explicitly linked in the provided text, but the demo page is available.
Limitations
The primary limitation is the reliance on in-the-wild data, which may contain inherent biases or quality issues despite filtering. The annotation of instructions and NV tags is limited to segments shorter than 5 minutes, which might restrict the granularity of control for very long, continuous expressive shifts. Additionally, the paper focuses heavily on Chinese and English for NV annotations, with other languages potentially having less expressive control support.
Broader impact
This dataset has significant potential to advance the field of long-form speech synthesis, enabling more realistic audiobooks, podcasts, and conversational agents. By providing a standardized benchmark with multi-speaker and long-duration support, it facilitates fair comparison of new architectures. The focus on controllable generation (style, NV) opens avenues for more expressive and human-like AI interactions. The paper introduces Emilia2, a 229K-hour multilingual dataset and benchmark for controllable long-form speech generation, demonstrating that models trained on this data achieve state-of-the-art performance in both long-form consistency and expressive control.
Author of this paper? Send a correction.