We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Primary: StepFun
All Institutions: StepFun
The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
The paper proposes a unified discrete autoregressive framework for general audio generation, departing from the dominant diffusion/flow-matching paradigm. The core architectural contribution is the "time-depth" decomposition: the LLM backbone predicts the first (semantic) RVQ codebook along the time axis, while a lightweight causal Transformer predicts the remaining 15 residual codebooks along the depth axis. This design effectively isolates the heavy acoustic modeling from the long-range linguistic planning, preventing the acoustic loss from overwhelming the LLM's text capabilities. The introduction of the "RVQ Adaptor," a zero-initialized residual module, is a clever engineering solution to align multi-codebook audio embeddings with the pretrained text embedding space without disrupting the backbone. The four-stage progressive pretraining curriculum, specifically the use of stop-gradient detachment for the residual predictor during the main generation phase, is a sophisticated approach to mitigating catastrophic forgetting and interference.
The evaluation is robust, covering both component-level ablations (RVQ Adaptor, progressive pretraining) and end-to-end generation quality. The ablation studies convincingly demonstrate that the RVQ Adaptor significantly improves audio understanding metrics (ASR, Translation) and that the progressive pretraining preserves text capabilities (MMLU, HumanEval) far better than a baseline. The TTS evaluation uses an Arena-style pairwise comparison with Elo ratings, showing strong performance against commercial systems. The Voice Design evaluation utilizes the InstructTTSEval benchmark with LLM-based judging (Gemini) and human preference testing, demonstrating state-of-the-art instruction following. The inclusion of subjective human evaluation for "human-likeness" is a strong point, as objective metrics often fail to capture prosodic naturalness.
The paper provides high-level architectural details and training recipes (learning rates, batch sizes, data mixtures) but lacks specific implementation details for the tokenizer training (e.g., specific GAN hyperparameters, discriminator architectures beyond "multi-period") and the exact composition of the 2.7T token pretraining corpus. The reliance on proprietary internal models (StepAudio R1.5, StepAudio Music Model) for data generation and evaluation limits full external reproducibility. However, the conceptual framework is clear enough for replication in principle.
The paper focuses heavily on TTS and Voice Design in its quantitative evaluation. While it claims capabilities in music, sound effects, and vocals, the rigorous benchmarking for these domains is less detailed compared to speech. The "Vibe Speech" and complex scene generation capabilities are demonstrated qualitatively but lack objective metrics. The model's performance on long-form generation (beyond the 44-minute context window mentioned) is not extensively tested. Additionally, the reliance on LLMs for reward modeling in RL (GRPO) introduces potential biases from the judge model.
This work significantly advances the field of unified audio generation by proving that discrete autoregressive models can compete with or surpass continuous diffusion models in speech tasks while offering better integration with LLMs. The "interference-aware" training strategy is a valuable lesson for the broader community of multimodal LLMs, providing a blueprint for adding new modalities without degrading existing capabilities. The unified instruction format (ROLE/DIRECTOR/SCRIPT) offers a practical standard for controlling complex audio scenes, which could influence future API designs and application development. The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Primary: Seoul National University
All Institutions: Seoul National University, KAIST, NAVER Cloud
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Primary: StepFun
All Institutions: StepFun
The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
The paper proposes a unified discrete autoregressive framework for general audio generation, departing from the dominant diffusion/flow-matching paradigm. The core architectural contribution is the "time-depth" decomposition: the LLM backbone predicts the first (semantic) RVQ codebook along the time axis, while a lightweight causal Transformer predicts the remaining 15 residual codebooks along the depth axis. This design effectively isolates the heavy acoustic modeling from the long-range linguistic planning, preventing the acoustic loss from overwhelming the LLM's text capabilities. The introduction of the "RVQ Adaptor," a zero-initialized residual module, is a clever engineering solution to align multi-codebook audio embeddings with the pretrained text embedding space without disrupting the backbone. The four-stage progressive pretraining curriculum, specifically the use of stop-gradient detachment for the residual predictor during the main generation phase, is a sophisticated approach to mitigating catastrophic forgetting and interference.
The evaluation is robust, covering both component-level ablations (RVQ Adaptor, progressive pretraining) and end-to-end generation quality. The ablation studies convincingly demonstrate that the RVQ Adaptor significantly improves audio understanding metrics (ASR, Translation) and that the progressive pretraining preserves text capabilities (MMLU, HumanEval) far better than a baseline. The TTS evaluation uses an Arena-style pairwise comparison with Elo ratings, showing strong performance against commercial systems. The Voice Design evaluation utilizes the InstructTTSEval benchmark with LLM-based judging (Gemini) and human preference testing, demonstrating state-of-the-art instruction following. The inclusion of subjective human evaluation for "human-likeness" is a strong point, as objective metrics often fail to capture prosodic naturalness.
The paper provides high-level architectural details and training recipes (learning rates, batch sizes, data mixtures) but lacks specific implementation details for the tokenizer training (e.g., specific GAN hyperparameters, discriminator architectures beyond "multi-period") and the exact composition of the 2.7T token pretraining corpus. The reliance on proprietary internal models (StepAudio R1.5, StepAudio Music Model) for data generation and evaluation limits full external reproducibility. However, the conceptual framework is clear enough for replication in principle.
The paper focuses heavily on TTS and Voice Design in its quantitative evaluation. While it claims capabilities in music, sound effects, and vocals, the rigorous benchmarking for these domains is less detailed compared to speech. The "Vibe Speech" and complex scene generation capabilities are demonstrated qualitatively but lack objective metrics. The model's performance on long-form generation (beyond the 44-minute context window mentioned) is not extensively tested. Additionally, the reliance on LLMs for reward modeling in RL (GRPO) introduces potential biases from the judge model.
This work significantly advances the field of unified audio generation by proving that discrete autoregressive models can compete with or surpass continuous diffusion models in speech tasks while offering better integration with LLMs. The "interference-aware" training strategy is a valuable lesson for the broader community of multimodal LLMs, providing a blueprint for adding new modalities without degrading existing capabilities. The unified instruction format (ROLE/DIRECTOR/SCRIPT) offers a practical standard for controlling complex audio scenes, which could influence future API designs and application development. The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Primary: Mila-Quebec AI Institute
All Institutions: Concordia University, Mila-Quebec AI Institute, Université Laval
ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
The paper proposes ZipCodec, a streaming neural speech codec that significantly reduces the frame rate to 6.25 Hz (160 ms per token) while maintaining a low bitrate of 0.80 kbps. The methodology is built upon the FocalCodec-Stream architecture but introduces several key modifications: (1) replacing the learned convolutional encoder with a causal log-mel frontend to simplify the pipeline; (2) introducing "ErfFormer," a transformer variant that removes RMSNorm and positional encodings, relying on DynamicErf activation and causal attention to handle long streaming contexts without positional extrapolation; (3) employing Scalar Spherical Quantization (SSQ) with a factorized bottleneck (64 dimensions, 4 levels) to achieve compact representation; and (4) a latency-aware streaming decoder that processes 8 WavLM representations jointly within the 160 ms window to relax causality constraints at the feature level. The training strategy involves large-scale distillation of WavLM layer-6 representations using ~94,000 hours of speech data, with specific noise and overlapping speech augmentations to match the teacher model's distribution.
The experimental evaluation is comprehensive, covering speech resynthesis (English and multilingual), voice conversion, and a suite of downstream discriminative and generative tasks (ASR, SI, SER, KS, IC, SE, SS). ZipCodec demonstrates superior performance compared to strong baselines like Mimi, PAST, and FocalCodec-Stream, particularly in intelligibility (dWER) and speaker similarity (Sim) at the significantly lower frame rate. The paper also provides detailed streaming efficiency metrics, showing real-time inference on consumer-grade CPUs and efficient GPU scaling. The results convincingly show that the 6.25 Hz bottleneck retains sufficient information for high-quality reconstruction and downstream tasks, narrowing the gap with non-streaming offline codecs.
The paper provides high reproducibility. It explicitly states that code and checkpoints are available. Detailed hyperparameters, training data sources (LibriLight, VoxPopuli, GigaSpeech), augmentation strategies, and architectural specifications (e.g., ErfFormer dimensions, SSQ parameters) are provided. The use of standard datasets and metrics (UTMOS, dWER, DNSMOS) allows for easy verification and comparison with other works in the field.
The primary limitation is the model size (842M parameters), which, while manageable for single-stream CPU inference, may be prohibitive for edge devices or high-concurrency server deployments compared to smaller codecs. Additionally, the evaluation is primarily focused on English and specific multilingual subsets; broader language coverage is not extensively tested. The reliance on WavLM distillation ties the codec's performance to the quality and biases of the WavLM teacher model.
This work has significant implications for speech-language modeling and real-time speech-to-speech systems. By reducing the frame rate to 6.25 Hz, the token sequence length is drastically shortened, which reduces computational costs and simplifies autoregressive modeling for speech LLMs. The low latency (160 ms) makes it suitable for highly responsive conversational AI applications. The open-sourcing of code and demos facilitates further research and adoption in the community. ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Primary: Imperial College London
All Institutions: Imperial College London, Technische Universität München, Mohamed bin Zayed University of Artificial Intelligence, Shanghai Jiao Tong University, Johns Hopkins University
The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
The paper proposes ARIA-Rubrics, a framework to evaluate the reasoning process of Large Audio Language Models (LALMs) rather than just final answer accuracy. The methodology relies on Chain-of-Thought (CoT) prompting to externalize reasoning into four steps: Perception, Analysis, Reasoning, and Answer. It introduces six metrics: Acoustic Grounding Score (using CLAP embeddings to verify entities against audio), Inter-step Coherence (using MPNet embeddings), Reasoning Chain Coverage (using a lightweight LLM judge for substantiveness), Reasoning Step Causality (using perplexity reduction), Reasoning Progress Score (using NLI entailment), and Audio Lexical Density (using Word2Vec similarity to AudioSet ontology). The approach is innovative in combining audio-specific perceptual checks with text-based reasoning quality metrics, addressing the gap where text-only metrics fail to detect perceptual hallucinations.
The authors evaluate 9 models across two benchmarks (MMAR and MMAU-mini). They identify three distinct reasoning modes: Reasoning, Scaffolding, and Decoration. The paper claims high correlation with human judgments, validating the automated metrics. The experimental setup is robust, covering both open-source and closed-source models, and the identification of specific failure modes (like scaffolding) provides actionable insights for model developers.
The code is available on GitHub. The paper specifies the models used for each metric (CLAP, MPNet, Qwen2.5-1.5B, RoBERTa-large-mnli, Word2Vec) and provides details on the CoT prompt structure and the AudioSet ontology filtering. This level of detail supports reproducibility, although the specific few-shot prompts for the LLM judge are referenced in appendices not fully visible in the truncated text, which is standard practice.
The framework relies on the quality of the CoT output; if a model refuses to generate CoT or generates unstructured text, the metrics may fail or be biased. The use of a small LLM (Qwen2.5-1.5B) for judging substantiveness and causality may limit the depth of evaluation compared to larger models. Additionally, the Audio Lexical Density metric is a proxy and may not capture all aspects of audio-relevant reasoning.
This work is significant for the development of reliable audio AI systems. By distinguishing between genuine reasoning and pattern matching, it provides a diagnostic tool for improving LALMs. It encourages the development of models that are not only accurate but also faithful in their reasoning, which is crucial for safety and interpretability in audio applications. The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
We present UniStream, a fully causal 48 kHz neural audio codec for streaming speech, music, and environmental sounds. At its core is Multi-Expert Residual Vector Quantization (ME-RVQ), which replaces the single shared codebook in each residual quantization layer with four expert codebooks controlled by a deterministic Top-K router. Because routing decisions are derived solely from previously decoded quantized states, the decoder can reproduce the selected experts without transmitting expert identifiers, thereby expanding quantization capacity while adding 5.5M parameters. We further introduce an auxiliary Optimal Transport Conditional Flow Matching (OT-CFM) objective to regularize the quantized latent space during training. The flow module is removed entirely at inference and therefore incurs no runtime overhead. UniStream supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode within a causal 48 kHz encoder-decoder framework, while achieving real-time GPU inference. To complement narrow-band speech metrics, we report 48 kHz ViSQOL audio mode, ViSQOL speech mode, standard VGGish-FAD, DNSMOS P.835, and higher-rate reference comparisons with Opus and EnCodec. At 12 kbps, UniStream-Top1 achieves PESQ and UTMOS scores comparable to EnCodec while reducing speech Mel-D from 13.07 to 8.21. At 22.5 kbps, UniStream-Top2 achieves a ViSQOL speech-mode score of 4.67 and an environmental audio-mode score of 3.96, exceeding all evaluated systems operating at 12 kbps or below in the latter setting. It also comes within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode. Ablation studies confirm that ME-RVQ is the primary source of quality improvement, whereas OT-CFM provides perceptual gains on speech with a mild trade-off in spectral distortion.
Primary: Tsinghua University
All Institutions: Tsinghua Shenzhen International Graduate School, Tsinghua University
UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
The paper proposes UniStream, a causal neural audio codec featuring Multi-Expert Residual Vector Quantization (ME-RVQ). The core innovation is replacing single codebooks in residual layers with four expert codebooks routed by a deterministic Top-K mechanism. Crucially, the routing is derived from previously decoded states, allowing the decoder to replicate expert selection without transmitting expert IDs, thus saving bandwidth. The method also introduces a training-only Optimal Transport Conditional Flow Matching (OT-CFM) regularizer to improve latent space quality without inference cost. The architecture is fully causal, suitable for streaming, and supports 12 kbps and 22.5 kbps modes.
Experiments are conducted on speech (LibriSpeech), music (MTG-Jamendo), and environmental audio (FSD50K). The evaluation is comprehensive, using both narrow-band metrics (PESQ, STOI, UTMOS) and full-band perceptual metrics (ViSQOL, VGGish-FAD, DNSMOS). Results show UniStream-Top1 outperforms EnCodec in spectral distortion (Mel-D) at 12 kbps, while UniStream-Top2 approaches Opus 24 kbps quality at a lower bitrate. Ablations confirm ME-RVQ is the primary driver of improvement, with OT-CFM providing modest perceptual gains.
The paper provides detailed architectural specifications (strides, dilation factors, codebook sizes, loss weights) and training hyperparameters. However, no code repository or demo link is provided in the text. The use of standard datasets and metrics facilitates reproduction, but the lack of open-source code limits immediate verification.
The paper lacks subjective listening tests (MOS), relying solely on objective metrics. Music domain performance (FAD) is noted as weaker compared to speech and environmental audio. The CPU inference time (RTF 1.05) is borderline for real-time deployment on standard hardware, though GPU inference is efficient. The comparison with non-causal baselines like EnCodec is somewhat unfair for streaming contexts, though acknowledged.
This work contributes to the field of low-latency neural audio coding, particularly for applications requiring full-band (48 kHz) fidelity in streaming scenarios. The deterministic routing mechanism offers a practical solution for expanding quantization capacity without signaling overhead, which could influence future codec designs. UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
Primary: KAIST
All Institutions: KAIST
The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
The paper proposes AV-STE, a modular front-end for enhancing semantic speech tokens in full-duplex dialogue systems. The core contribution is the decoupling of robustness enhancement from the large language model (LLM) backbone. By keeping the speech LLM (Moshi) frozen and only enhancing the semantic token stream using a streaming audio-visual encoder (modified AV-HuBERT) and a noise-adaptive fusion module, the authors address a significant practical limitation of existing multimodal dialogue systems that require costly retraining. The introduction of "Soft Token Cross-Attention" to handle uncertainty in noisy tokenizer outputs and "Noise-Adaptive Modulation" based on entropy are technically sound and well-motivated innovations. The adaptation of bidirectional AV-HuBERT to a causal, streaming architecture with bounded lookahead is a necessary and well-executed engineering contribution for real-time applications.
The experimental setup is rigorous, covering in-domain (LRS3) and out-of-domain (Seamless Interaction) datasets, with various noise types (non-speech, speaker interference). The evaluation metrics are appropriate for the task: semantic token accuracy for the front-end and GPT-4o-judged coherence plus turn-taking metrics (TOR, latency) for the downstream dialogue. The results show substantial improvements in token recovery (e.g., from 12.61% to 65.60% under same-dataset interference) and downstream coherence (1.42 to 1.91). The comparison against a non-streaming baseline (PASE) and a different dialogue model (PersonaPlex) provides useful context, although the primary comparison is against the frozen Moshi baseline. The qualitative examples strongly support the quantitative findings.
The paper provides a GitHub link to the code and models. The training configuration, dataset preprocessing, and model architecture details are described in sufficient detail for reproduction. The use of standard datasets (LRS3, AudioSet, Seamless Interaction) and open-source components (Moshi, AV-HuBERT) further enhances reproducibility.
The paper acknowledges that AV-STE assumes reliable visual input and does not handle visual occlusion or missing video. There is a noted domain shift issue where clean-token accuracy drops on out-of-domain data (Seamless Interaction), indicating sensitivity to domain characteristics. The reliance on GPT-4o for evaluation, while common, introduces a potential bias, though it is a standard practice in current LLM evaluation. The method is specific to the Moshi/Mimi tokenization scheme, limiting immediate applicability to other speech LLM architectures without adaptation.
This work has significant implications for the deployment of robust spoken dialogue systems in real-world, noisy environments. By providing a plug-and-play module that enhances robustness without retraining the expensive LLM backbone, it lowers the barrier to entry for audio-visual dialogue systems. The focus on full-duplex interaction is timely and relevant to the next generation of conversational AI. The approach could be extended to other modalities or tokenization schemes, potentially influencing the design of future multimodal speech interfaces. The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Primary: Tencent
All Institutions: Tencent, Zhejiang University, Shanghai Jiao Tong University, The Chinese University of Hong Kong, Nanyang Technological University
Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
The paper proposes "Gander," an end-to-end Omni Interaction Agent that unifies perception, real-time interaction, and agentic capabilities. The core architectural innovation is the "Cerebellum-Brain" collaborative framework. The "Cerebellum" is a streaming Thinker-Talker model responsible for low-latency, full-duplex interaction, while the "Brain" is a training-free, plug-and-play agent (e.g., Codex, Claude Code) handling complex reasoning. The Cerebellum uses a "streaming chunk flattening" mechanism, serializing audio, video, and text into a unified token stream where the model predicts control tokens (listen/speak/interrupt) at each 1-second chunk. This allows for native interruption and proactive feedback without external VAD modules. The methodology is sound, addressing the latency-intelligence trade-off by decoupling fast interaction from slow reasoning.
The evaluation is comprehensive, covering conversational ability, omni understanding, interactive capability, and agentic intelligence. The paper reports internal human evaluations showing Gander maintains SOTA spoken dialogue naturalness while achieving competitive omni interaction performance. It demonstrates robustness in noisy environments, multi-party interactions, and backchannel communication. However, the paper relies heavily on "internal human evaluations" and subjective demos, with limited quantitative benchmarks compared to specific baselines (e.g., GPT-4o, MiniCPM-o) in the provided text snippet. The lack of standardized public benchmark scores for the "agentic" component makes direct comparison difficult.
The authors state they release the model weights, code, and data. The architecture details (chunk size, token rates, encoder types like SigLIP) are provided. However, the "Brain" component relies on proprietary external agents (Codex, Claude Code), which limits full reproducibility of the agentic performance. The training data pipeline is described in detail, including the synthesis of full-duplex interaction data, which aids reproducibility of the Cerebellum component.
The system's performance is coupled to the quality of the external "Brain" agent, which may have its own latency and cost implications. The "training-free" nature of the Brain means the Cerebellum cannot learn from the Brain's specific failures or adapt its tool-calling strategy through end-to-end fine-tuning. The evaluation lacks extensive quantitative ablation studies comparing the Cerebellum-Brain architecture against monolithic models on standardized agentic benchmarks.
This work is significant for the development of natural, human-like AI assistants that can handle complex tasks while maintaining fluid conversation. The decoupled architecture offers a scalable path for integrating powerful reasoning models into real-time interfaces. It pushes the field beyond turn-based chatbots toward continuous, multimodal collaboration. Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Primary: Seoul National University
All Institutions: Seoul National University, KAIST, NAVER Cloud
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.