WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
Abstract
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral $v$-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
WaveGen proposes a fully end-to-end text-to-waveform diffusion model using Spectral Forcing to jointly learn internal spectral and waveform trajectories, achieving state-of-the-art zero-shot TTS performance without external vocoders or duration predictors. The paper provides rigorous ablation studies on prediction/loss spaces and scalability, demonstrating that direct waveform generation can match or exceed multi-stage pipelines when properly optimized with decoupled $x$-prediction and $v$-loss objectives.
Methodology
The paper proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework. The core methodological contribution is "Spectral Forcing," which jointly learns an internal low-resolution spectral trajectory (Mel-spectrogram) and the target high-resolution waveform trajectory within a single DiT architecture. This approach avoids the need for external vocoders, codecs, or duration predictors. The model utilizes a decoupled prediction and loss space strategy: it predicts the clean signal ($x$-prediction) to ensure accurate energy distribution but computes the loss in the velocity space ($v$-loss) to maintain training stability. Additionally, a multi-scale spectral $v$-loss is introduced to capture time-frequency structures at various resolutions. The architecture employs shared DiT blocks for alignment and disentangled heads for representation-specific refinement.
Results & evaluation
The experiments are extensive, covering scalability (0.1B to 0.6B parameters), dataset scale (LibriTTS to Emilia 120k hours), and ablation studies on key components (Spectral Forcing, prediction/loss spaces, disentangled heads). The model achieves state-of-the-art results on the Seed-en benchmark with a WER of 1.47 and SIM of 0.75, outperforming complex multi-stage pipelines. Subjective evaluations (MOS) via MTurk confirm comparable naturalness and similarity to strong baselines like F5-TTS and CosyVoice3. The paper also provides insights into NFE (Number of Function Evaluations) and CFG (Classifier-Free Guidance) effects.
Reproducibility
The authors provide source code and checkpoints on GitHub. Detailed hyperparameters, training configurations, and implementation details are included in the appendix. The use of open-source datasets (LibriTTS, Emilia) and standard evaluation metrics (Whisper for WER, WavLM for SIM, DNSMOS) enhances reproducibility.
Limitations
The model is limited to 24 kHz sampling rate, whereas state-of-the-art systems often support 48 kHz. The training was restricted to 4 epochs for most models due to computational constraints, though trends suggest further improvement with longer training. The reliance on MP3-compressed Emilia data may limit high-frequency detail generation compared to uncompressed data.
Broader impact
This work demonstrates that fully end-to-end waveform generation is a viable alternative to complex cascaded TTS pipelines, potentially simplifying deployment and reducing error propagation from intermediate stages. It contributes to the understanding of diffusion models in high-dimensional audio spaces and offers a scalable framework for future zero-shot TTS research. WaveGen proposes a fully end-to-end text-to-waveform diffusion model using Spectral Forcing to jointly learn internal spectral and waveform trajectories, achieving state-of-the-art zero-shot TTS performance without external vocoders or duration predictors. The paper provides rigorous ablation studies on prediction/loss spaces and scalability, demonstrating that direct waveform generation can match or exceed multi-stage pipelines when properly optimized with decoupled $x$-prediction and $v$-loss objectives.
Author of this paper? Send a correction.