Walrus Technical Report
Abstract
We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding. Our main contribution is not a new architecture but a data-centric training recipe; we use a standard decoder-only backbone initialized from Gemma 4, so that gains can be traced to the data. For speech generation, Walrus predicts low-bitrate discrete speech tokens, and a lightweight token-to-waveform decoder converts them into high-fidelity audio. The recipe is a three-stage data curriculum that moves from broad multilingual data to multi-talker, code-switching, and domain-specific data. At each stage, task-aware and dataset-aware weighting balances tasks with very different token budgets and protects core skills from forgetting. Trained on over 600M samples covering more than 20 tasks and 100 languages, Walrus leads on most multilingual, multi-talker, and code-switching ASR, speech translation, and audio understanding benchmarks, outperforming both open speech language models of comparable size and dedicated task-specific models. On most benchmarks, it even matches or surpasses the $4\times$ larger Qwen3-Omni's performance. Results suggest that data engineering is a primary driver of quality in multi-task speech and audio modeling.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution is a data-centric training recipe for a unified 8B speech-and-audio foundation model that achieves state-of-the-art results across diverse tasks without architectural novelty. By fixing the backbone and focusing on a three-stage data curriculum with task-aware weighting, the paper demonstrates that data engineering is a primary driver of quality in multi-task speech modeling, outperforming larger models and dedicated task-specific systems on key benchmarks for ASR, translation, and audio understanding.
Methodology
The paper proposes a data-centric approach to training a unified speech-and-audio foundation model, explicitly arguing that data engineering is more critical than architectural novelty. The methodology relies on a standard decoder-only Transformer backbone (Gemma 4 E4B) initialized with pre-trained weights, keeping the architecture fixed to isolate the impact of the data recipe. The core innovation lies in a three-stage data curriculum: Stage 1 establishes a broad multilingual foundation with high weights for ASR; Stage 2 introduces complex tasks like multi-talker ASR, code-switching, and timestamp prediction while reducing weights of foundation data to prevent catastrophic forgetting; Stage 3 focuses on domain-specific and specialized data. A key technical contribution is the use of task-aware and dataset-aware weighting to balance the disparate token budgets of generative tasks (TTS/S2ST) versus discriminative tasks (ASR), preventing generative tasks from dominating the loss. For speech generation, the model predicts low-bitrate discrete tokens (325 bps) using a VQ tokenizer trained on W2v-BERT features, which are then decoded into waveforms by a GAN-based vocoder conditioned on source speaker features for identity preservation.
Results & evaluation
The experimental evaluation is extensive, covering over 20 tasks and 100 languages. Walrus demonstrates state-of-the-art performance on multilingual ASR (VoxPopuli, FLEURS), code-switching ASR (CS-FLEURS), and multi-talker ASR (LibriMix, Daily Talk), significantly outperforming comparable open models and even matching or surpassing the much larger Qwen3-Omni (30B) on several benchmarks. Notably, it achieves a cpWER of 19.32 on LibriMix clean 2-mix, which is half the error rate of Qwen3-Omni. The paper also evaluates speech translation (S2ST) and audio understanding (MMSU, MMAU-Pro), showing strong results. The comparison between the Commercial-Ready (CR) and Research (R) variants provides insight into the impact of data licensing and quality, with the R variant showing consistent improvements across all benchmarks.
Reproducibility
The paper provides a high level of detail regarding the training setup, including optimizer hyperparameters, batch sizes, and the specific data weighting schemes for each stage. It lists the specific public datasets used (CommonVoice, VoxPopuli, LibriSpeech, etc.) and describes the synthetic data generation pipelines for code-switching and multi-talker scenarios. However, as is common with large-scale industrial models, the exact composition of the "internal data" and the specific implementation details of the proprietary data cleaning pipelines are not fully disclosed. The code and model weights are available via the provided GitHub link, which enhances reproducibility for those with access to the resources.
Limitations
The primary limitation is the reliance on a fixed backbone (Gemma 4 E4B), which may limit the model's scalability compared to architectures designed specifically for multimodal alignment. The paper does not provide a detailed ablation study isolating the contribution of each individual stage of the curriculum or the specific impact of the task-weighting strategy versus the curriculum order alone. Additionally, the evaluation of audio understanding is somewhat limited compared to the extensive ASR and translation results, and there is no subjective evaluation (e.g., MOS) for the generated speech, relying instead on objective metrics and comparisons to baselines.
Broader impact
This work has significant implications for the deployment of unified speech models in real-world applications, demonstrating that a single model can handle a wide range of tasks (transcription, translation, generation, understanding) efficiently. The emphasis on data engineering provides a roadmap for other researchers and companies to improve their speech models without necessarily increasing model size or complexity. The availability of a commercial-ready variant trained on licensed data is particularly valuable for industry adoption. The results suggest that careful data curation and mixing can yield performance gains comparable to or exceeding those from architectural changes, potentially shifting research focus towards data-centric methods in speech AI. The main contribution is a data-centric training recipe for a unified 8B speech-and-audio foundation model that achieves state-of-the-art results across diverse tasks without architectural novelty. By fixing the backbone and focusing on a three-stage data curriculum with task-aware weighting, the paper demonstrates that data engineering is a primary driver of quality in multi-task speech modeling, outperforming larger models and dedicated task-specific systems on key benchmarks for ASR, translation, and audio understanding.
Author of this paper? Send a correction.