We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Primary: StepFun
All Institutions: StepFun
The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
The paper proposes a unified discrete autoregressive framework for general audio generation, departing from the dominant diffusion/flow-matching paradigm. The core architectural contribution is the "time-depth" decomposition: the LLM backbone predicts the first (semantic) RVQ codebook along the time axis, while a lightweight causal Transformer predicts the remaining 15 residual codebooks along the depth axis. This design effectively isolates the heavy acoustic modeling from the long-range linguistic planning, preventing the acoustic loss from overwhelming the LLM's text capabilities. The introduction of the "RVQ Adaptor," a zero-initialized residual module, is a clever engineering solution to align multi-codebook audio embeddings with the pretrained text embedding space without disrupting the backbone. The four-stage progressive pretraining curriculum, specifically the use of stop-gradient detachment for the residual predictor during the main generation phase, is a sophisticated approach to mitigating catastrophic forgetting and interference.
The evaluation is robust, covering both component-level ablations (RVQ Adaptor, progressive pretraining) and end-to-end generation quality. The ablation studies convincingly demonstrate that the RVQ Adaptor significantly improves audio understanding metrics (ASR, Translation) and that the progressive pretraining preserves text capabilities (MMLU, HumanEval) far better than a baseline. The TTS evaluation uses an Arena-style pairwise comparison with Elo ratings, showing strong performance against commercial systems. The Voice Design evaluation utilizes the InstructTTSEval benchmark with LLM-based judging (Gemini) and human preference testing, demonstrating state-of-the-art instruction following. The inclusion of subjective human evaluation for "human-likeness" is a strong point, as objective metrics often fail to capture prosodic naturalness.
The paper provides high-level architectural details and training recipes (learning rates, batch sizes, data mixtures) but lacks specific implementation details for the tokenizer training (e.g., specific GAN hyperparameters, discriminator architectures beyond "multi-period") and the exact composition of the 2.7T token pretraining corpus. The reliance on proprietary internal models (StepAudio R1.5, StepAudio Music Model) for data generation and evaluation limits full external reproducibility. However, the conceptual framework is clear enough for replication in principle.
The paper focuses heavily on TTS and Voice Design in its quantitative evaluation. While it claims capabilities in music, sound effects, and vocals, the rigorous benchmarking for these domains is less detailed compared to speech. The "Vibe Speech" and complex scene generation capabilities are demonstrated qualitatively but lack objective metrics. The model's performance on long-form generation (beyond the 44-minute context window mentioned) is not extensively tested. Additionally, the reliance on LLMs for reward modeling in RL (GRPO) introduces potential biases from the judge model.
This work significantly advances the field of unified audio generation by proving that discrete autoregressive models can compete with or surpass continuous diffusion models in speech tasks while offering better integration with LLMs. The "interference-aware" training strategy is a valuable lesson for the broader community of multimodal LLMs, providing a blueprint for adding new modalities without degrading existing capabilities. The unified instruction format (ROLE/DIRECTOR/SCRIPT) offers a practical standard for controlling complex audio scenes, which could influence future API designs and application development. The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
Primary: Alibaba Group
All Institutions: Alibaba Group, Shanghai Jiao Tong University
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
The paper introduces DiffSynth-Music, a framework for controllable music generation built upon a frozen Diffusion Transformer (DiT) backbone (ACE-Step-1.5-XL-SFT). The core methodology involves injecting layer-wise key-value (KV) representations derived from audio conditioning signals into the generation branch. This approach is conceptually related to adapter-based methods like ControlNet but is tailored for audio and uses a shared VAE to map diverse conditioning waveforms into a common latent space. A key strength is the design of three template models (Control, Prosody, Reference) that support five distinct control types: beats, vocals, accompaniment, prosody, and reference audio. The detailed construction of these control signals from raw music recordings (e.g., beat extraction, source separation, vocal resynthesis for prosody, loudest excerpt for reference) is well-described and practical. The prosody generation method, which uses a sinusoidal carrier with estimated pitch and envelope to reduce phonetic/timbral cues, is a clever way to achieve melodic control without enforcing specific vocal identity. The KV-injection mechanism, where control keys and values are concatenated with the generation branch's keys and values, allows for composable conditioning. The efficiency gain from computing each control cache once at a fixed template timestep (clean-data endpoint) and reusing it throughout sampling is a significant practical advantage, reducing computational overhead. The training formulation uses conditional flow matching, which is a suitable choice for learning continuous-time vector fields, and the adapter-only optimization ensures the pretrained backbone remains frozen, preserving its general generation capabilities. The overall architecture is sound, well-motivated, and addresses a clear need in controllable music generation.
The experimental evaluation is comprehensive and well-structured. The authors evaluate DiffSynth-Music on both Mandarin and English songs, demonstrating its applicability across different languages, which is a strong point. The comparison against several state-of-the-art music generation systems (ACE-Step-1.5-XL-SFT, DiffRhythm-2, HeartMuLa-3B, MiniMax-Music3, LeVo-2-Large) provides a robust baseline. Critically, the backbone model (ACE-Step-1.5-XL-SFT) is included as a baseline to specifically highlight the improvements gained by adding the audio-conditioned templates. The evaluation metrics are appropriate and cover both control adherence and general music quality. For control adherence, metrics like Beat-F1, Cemgil (beats), V-MSE, PER (vocals), A-MSE (accompaniment), Pitch_50 (prosody), and MuLan-A (reference) are used. The results clearly show that DiffSynth-Music significantly outperforms all baselines across all seven control adherence metrics, often by a large margin (e.g., Beat-F1 from ~0.3 to ~0.85, Pitch_50 from ~0.04 to ~0.47). This strongly validates the effectiveness of the proposed conditioning framework. To assess general music quality, AudioBox-Aesthetics scores (CE, CU, PC, PQ) and MuLan-T (text-music alignment) are used. The results indicate that DiffSynth-Music maintains "broadly comparable" scores to the evaluated base models, with some metric-specific trade-offs. While some scores are slightly lower than the backbone, the overall message is that the added controllability does not severely degrade the general generation quality, which is a reasonable and expected outcome for such an adapter-based system. The experimental setup and results convincingly support the paper's claims.
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
Realtime spoken interaction demands deep reasoning, prompt responses, and fluid turn-taking. We present StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop. Deep Perception captures rich acoustic cues to interpret user intent, while Seamless Duplex models synchronized audio streams to handle pauses, backchannels, and interruptions naturally. Crucially, we resolve the tension between deep deliberation and latency via Think-While-Speaking, executing private reasoning in parallel with spoken delivery. In reasoning mode, StepAudio 3 reaches a 73.0 macro average on StepAudioChat. With Think-While-Speaking, it achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time. Furthermore, an integrated Voice Agent handles asynchronous tool execution without disrupting the dialogue flow. StepAudio 3 Realtime achieves top-tier performance across key dimensions: an exceptional 90.6 on the MMSU benchmark, 98.9 Overall on the Artificial Analysis Full-Duplex Bench, and a 56.0% macro task-success rate on $τ$-Voice.
Primary: StepFun
All Institutions: StepFun
StepAudio 3 Realtime introduces a "Think-While-Speaking" architecture that enables parallel reasoning and speech generation, achieving state-of-the-art performance in full-duplex interaction and audio understanding while maintaining competitive reasoning capabilities. The paper demonstrates a robust system design that balances latency, intelligence, and conversational fluidity, marking a significant milestone in the development of real-time audio-language foundation models.
The paper proposes a comprehensive architecture for real-time spoken interaction, centered on a "listen-converse-think-act" loop. The core technical contribution is the "Think-While-Speaking" mechanism, which decouples the formulation of reasoning (Formulation Brain) from the articulation of speech (Articulation Brain), allowing the model to speak while continuing to reason in the background. This is supported by "Adaptive Thinking," a routing mechanism that determines whether a turn requires explicit reasoning based on a blind judge's evaluation of the necessity of thought, and Multi-Token Prediction (MTP) for accelerating private reasoning. The system also features "Seamless Duplex" for full-duplex interaction, handling interruptions and backchannels via time-interleaved audio blocks and state tokens. The model utilizes a Mixture-of-Experts (MoE) architecture with 196B total parameters (11B active) and integrates a Voice Agent for asynchronous tool execution. The methodology is sophisticated, combining architectural innovations with specific training strategies like model merging of specialized teachers to balance audio understanding, text reasoning, and dialogue capabilities.
The experimental evaluation is extensive, covering ASR (LibriSpeech, AISHELL, WenetSpeech), audio understanding (MMSU, MMAU, MMAR, etc.), full-duplex interaction (Artificial Analysis Full-Duplex Bench), and agentic tasks ($\tau$-Voice). The model achieves state-of-the-art or competitive results in most categories, notably leading in MMSU (90.6) and Full-Duplex Bench (98.9). However, it trails competitors like Kimi K3 in pure reasoning/dialogue quality (73.0 vs 77.1 macro average) and GPT-Realtime in retail agentic tasks. The ablation studies on Adaptive Thinking and MTP provide insight into the trade-offs between latency, quality, and reasoning depth. The "Less is more" ablation on data quality is particularly strong, showing significant gains from high-quality SFT data over volume.
The paper provides high-level architectural details and training stages but lacks specific hyperparameters, exact data composition percentages, and code availability. The use of proprietary benchmarks (StepAudioChat) and internal data pipelines limits external reproducibility. The model merging weights (3:1:1:1) are specified, which aids in understanding the integration strategy, but the specific teacher training data is not public.
The model underperforms compared to top-tier LLMs (like Kimi K3) in pure reasoning and dialogue quality when not in real-time mode. The "Adaptive Thinking" mechanism shows inconsistent benefits, sometimes reducing reasoning scores compared to full thinking. The agentic performance in retail domains is notably weak (37.7%). The paper is a technical report without peer review, and some benchmarks are proprietary or newly introduced by the authors, making independent verification difficult.
This work significantly advances the field of real-time voice assistants by addressing the latency-reasoning trade-off. The "Think-While-Speaking" paradigm is a crucial step towards more intelligent, human-like voice interactions that can handle complex tasks without sacrificing responsiveness. The integration of full-duplex capabilities with agentic tool use sets a new standard for voice AI systems. StepAudio 3 Realtime introduces a "Think-While-Speaking" architecture that enables parallel reasoning and speech generation, achieving state-of-the-art performance in full-duplex interaction and audio understanding while maintaining competitive reasoning capabilities. The paper demonstrates a robust system design that balances latency, intelligence, and conversational fluidity, marking a significant milestone in the development of real-time audio-language foundation models.
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.
Primary: Ant Group
All Institutions: Ant Group, Tsinghua University
Realtime-Venus introduces a dual-loop runtime architecture that enables proactive full-duplex multimodal interaction with asynchronous tool delegation, achieving state-of-the-art results in streaming video understanding and conversational continuity metrics. The system effectively bridges the gap between real-time perception and complex background reasoning, offering a scalable framework for next-generation spoken and visual agents.
The paper proposes Realtime-Venus, a system built on the MiniCPM-o 4.5 architecture, featuring two 9B models (Omni and Audio) for full-duplex interaction. The core methodological contribution is the "Dual-Loop" runtime: a foreground loop for latency-sensitive perception and speech generation, and a background "Harness" loop for asynchronous tool execution and reasoning. The models are trained to emit private `
The evaluation is extensive, covering streaming video understanding (StreamingBench, OVO-Bench), audio understanding (MMAU, MMAU-Pro), and specifically full-duplex metrics (Full-Duplex-Bench v1.5). The results show Realtime-Venus-Omni leading on 6/8 video benchmarks and Realtime-Venus-Audio performing competitively on audio tasks. Notably, the full-duplex metrics show high continuation rates (97% under backchannels), demonstrating the system's ability to distinguish between non-interruptive and interruptive speech. However, the comparison is limited to a specific set of online models, and the "delegation" success is measured via an internal benchmark, limiting external validation of the tool-use component.
The paper provides detailed descriptions of the architecture, data pipeline, and training recipe. However, as a system paper from a large industry lab (Ant Group), specific hyperparameters, exact data construction scripts, and the code for the "Harness" runtime are not fully detailed in a way that would allow independent reproduction without significant reverse engineering. The reliance on the open-source MiniCPM-o 4.5 base model helps, but the proprietary data synthesis pipeline is a black box.
The primary limitation is the lack of open-source code and data for the specific Realtime-Venus models and the Harness framework. The evaluation of the asynchronous delegation component relies on an internal benchmark, making it difficult for the community to verify the quality of tool-use integration. Additionally, the paper does not provide ablation studies isolating the contribution of the memory module vs. the base model improvements in the main video benchmarks.
This work is significant for the development of real-time AI assistants that can handle complex, multi-step tasks while maintaining natural conversational flow. The dual-loop architecture addresses a critical bottleneck in current LLM agents: the latency of tool execution. By decoupling perception/speech from reasoning/tools, it enables more robust and human-like interaction patterns, particularly in multimodal environments. Realtime-Venus introduces a dual-loop runtime architecture that enables proactive full-duplex multimodal interaction with asynchronous tool delegation, achieving state-of-the-art results in streaming video understanding and conversational continuity metrics. The system effectively bridges the gap between real-time perception and complex background reasoning, offering a scalable framework for next-generation spoken and visual agents.
Asked to describe what one of six speakers in a recording talks about, audio language models describe the right one on 6 to 16% of trials, below the 16.7% a guess would give. Adding a fixed bias to the attention logits of a hundred heads, under a tenth of the model's and with no training, redirects the description to whichever speaker we choose, on 90.7% to 99.0% of trials. Those heads are largely not specific to audio. Rank the text-only language model an audio model was built from, or a released model of the same family, on a written version of the task, take its top hundred heads, and carry them over unchanged: they redirect the audio model on 80.8% to 95.0% of trials, with nothing about audio entering the selection. The audio and text head sets share 66 to 74 of 100 where chance would give about 20, and the shared part alone reproduces almost all of the steering. What that does not show is that sharing is what makes the heads work: an equal-sized draw from the same discovered hundred does nearly as well, and none of our three models separates the two explanations. A second finding concerns how such heads are found. Ranking heads by how much attention they place on the segment asked about, as an established score does, or by how much of their attention moves with the question, as a per-head normalised variant does, gives top hundreds that share 69, 37 and 4 heads across our three models. In Ultravox, where they share 4, the established score's heads leave output the judge cannot place on any segment on 69.7% of trials, against 40.0% with no intervention and 1.0% for the normalised variant. That is one arm of six; on the other five the established score steers above a random draw.
Primary: Cornell University
All Institutions: Cornell University
The paper demonstrates that audio language models inherit speaker-tracking capabilities from their text backbones, as evidenced by the ability of text-only ranked attention heads to effectively steer audio model outputs via simple logit biasing. This finding challenges the assumption of modality-specific mechanisms for high-level audio understanding and provides a rigorous causal framework for analyzing attention heads in multimodal systems, revealing that standard attention mass scores can be misleading compared to selectivity-based metrics.
The paper proposes a rigorous mechanistic analysis of audio language models (ALMs) using attention head interventions. The core method involves identifying "tracking heads" that attend to specific speaker segments and applying a fixed bias to their attention logits to steer the model's output. The novelty lies in the comparison between heads selected via audio-specific metrics (mass vs. selectivity) and heads selected purely from the text-only backbone. The authors demonstrate that text-derived head rankings can effectively steer audio models, suggesting that speaker tracking mechanisms are inherited from the language backbone rather than being modality-specific. The methodology is sound, utilizing causal interventions (biasing logits) rather than mere correlation, and includes careful controls for random head selection and layer matching.
The experiments are conducted on three distinct ALMs (Qwen2-Audio, Ultravox, SALMONN) using a custom dataset of 6-speaker recordings. The evaluation metrics include accuracy of speaker attribution and "unattributable rate" (abstention). The results show that the proposed intervention significantly improves attribution accuracy (from ~10% to >90%) and that text-derived heads are largely sufficient for this task. The comparison between mass and selectivity rankings reveals that the standard "mass" score can lead to unattributable outputs in some models, highlighting the importance of the selectivity metric. The inclusion of random controls and layer-matched baselines strengthens the causal claims.
The paper provides detailed descriptions of the intervention (bias magnitude, token spans) and the scoring metrics. However, the specific implementation details for the "lexical judge" and the exact prompt templates are not fully detailed in the provided text, which might hinder exact reproduction. The use of 4-bit quantization is noted as a potential threat to fidelity, which is a good transparency point. The code and data availability are not explicitly stated in the provided text, but the methodology is described with enough precision for a skilled researcher to attempt reproduction.
The primary limitation is the reliance on a lexical judge for evaluation, which may not capture all nuances of speaker attribution. The paper acknowledges that the "sharing" of heads between text and audio models does not prove causality of the sharing itself, as random draws from the same set also work. The analysis is limited to three models, and the generalizability to other architectures or tasks is not fully established. The use of a fixed bias magnitude ($B=10^4$) for comparability might not reflect natural model behavior, though $B=5$ is also tested.
This work has significant implications for the interpretability of multimodal models, suggesting that high-level reasoning tasks like speaker tracking may rely on pre-existing linguistic structures rather than modality-specific features. This could inform the design of more efficient multimodal models and provide insights into how attention mechanisms generalize across modalities. It also highlights the limitations of standard attention scoring metrics (mass) for causal analysis, advocating for more nuanced metrics like selectivity. The paper demonstrates that audio language models inherit speaker-tracking capabilities from their text backbones, as evidenced by the ability of text-only ranked attention heads to effectively steer audio model outputs via simple logit biasing. This finding challenges the assumption of modality-specific mechanisms for high-level audio understanding and provides a rigorous causal framework for analyzing attention heads in multimodal systems, revealing that standard attention mass scores can be misleading compared to selectivity-based metrics.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Primary: StepFun
All Institutions: StepFun
The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
The paper proposes a unified discrete autoregressive framework for general audio generation, departing from the dominant diffusion/flow-matching paradigm. The core architectural contribution is the "time-depth" decomposition: the LLM backbone predicts the first (semantic) RVQ codebook along the time axis, while a lightweight causal Transformer predicts the remaining 15 residual codebooks along the depth axis. This design effectively isolates the heavy acoustic modeling from the long-range linguistic planning, preventing the acoustic loss from overwhelming the LLM's text capabilities. The introduction of the "RVQ Adaptor," a zero-initialized residual module, is a clever engineering solution to align multi-codebook audio embeddings with the pretrained text embedding space without disrupting the backbone. The four-stage progressive pretraining curriculum, specifically the use of stop-gradient detachment for the residual predictor during the main generation phase, is a sophisticated approach to mitigating catastrophic forgetting and interference.
The evaluation is robust, covering both component-level ablations (RVQ Adaptor, progressive pretraining) and end-to-end generation quality. The ablation studies convincingly demonstrate that the RVQ Adaptor significantly improves audio understanding metrics (ASR, Translation) and that the progressive pretraining preserves text capabilities (MMLU, HumanEval) far better than a baseline. The TTS evaluation uses an Arena-style pairwise comparison with Elo ratings, showing strong performance against commercial systems. The Voice Design evaluation utilizes the InstructTTSEval benchmark with LLM-based judging (Gemini) and human preference testing, demonstrating state-of-the-art instruction following. The inclusion of subjective human evaluation for "human-likeness" is a strong point, as objective metrics often fail to capture prosodic naturalness.
The paper provides high-level architectural details and training recipes (learning rates, batch sizes, data mixtures) but lacks specific implementation details for the tokenizer training (e.g., specific GAN hyperparameters, discriminator architectures beyond "multi-period") and the exact composition of the 2.7T token pretraining corpus. The reliance on proprietary internal models (StepAudio R1.5, StepAudio Music Model) for data generation and evaluation limits full external reproducibility. However, the conceptual framework is clear enough for replication in principle.
The paper focuses heavily on TTS and Voice Design in its quantitative evaluation. While it claims capabilities in music, sound effects, and vocals, the rigorous benchmarking for these domains is less detailed compared to speech. The "Vibe Speech" and complex scene generation capabilities are demonstrated qualitatively but lack objective metrics. The model's performance on long-form generation (beyond the 44-minute context window mentioned) is not extensively tested. Additionally, the reliance on LLMs for reward modeling in RL (GRPO) introduces potential biases from the judge model.
This work significantly advances the field of unified audio generation by proving that discrete autoregressive models can compete with or surpass continuous diffusion models in speech tasks while offering better integration with LLMs. The "interference-aware" training strategy is a valuable lesson for the broader community of multimodal LLMs, providing a blueprint for adding new modalities without degrading existing capabilities. The unified instruction format (ROLE/DIRECTOR/SCRIPT) offers a practical standard for controlling complex audio scenes, which could influence future API designs and application development. The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
Primary: Alibaba Group
All Institutions: Alibaba Group, Shanghai Jiao Tong University
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
The paper introduces DiffSynth-Music, a framework for controllable music generation built upon a frozen Diffusion Transformer (DiT) backbone (ACE-Step-1.5-XL-SFT). The core methodology involves injecting layer-wise key-value (KV) representations derived from audio conditioning signals into the generation branch. This approach is conceptually related to adapter-based methods like ControlNet but is tailored for audio and uses a shared VAE to map diverse conditioning waveforms into a common latent space. A key strength is the design of three template models (Control, Prosody, Reference) that support five distinct control types: beats, vocals, accompaniment, prosody, and reference audio. The detailed construction of these control signals from raw music recordings (e.g., beat extraction, source separation, vocal resynthesis for prosody, loudest excerpt for reference) is well-described and practical. The prosody generation method, which uses a sinusoidal carrier with estimated pitch and envelope to reduce phonetic/timbral cues, is a clever way to achieve melodic control without enforcing specific vocal identity. The KV-injection mechanism, where control keys and values are concatenated with the generation branch's keys and values, allows for composable conditioning. The efficiency gain from computing each control cache once at a fixed template timestep (clean-data endpoint) and reusing it throughout sampling is a significant practical advantage, reducing computational overhead. The training formulation uses conditional flow matching, which is a suitable choice for learning continuous-time vector fields, and the adapter-only optimization ensures the pretrained backbone remains frozen, preserving its general generation capabilities. The overall architecture is sound, well-motivated, and addresses a clear need in controllable music generation.
The experimental evaluation is comprehensive and well-structured. The authors evaluate DiffSynth-Music on both Mandarin and English songs, demonstrating its applicability across different languages, which is a strong point. The comparison against several state-of-the-art music generation systems (ACE-Step-1.5-XL-SFT, DiffRhythm-2, HeartMuLa-3B, MiniMax-Music3, LeVo-2-Large) provides a robust baseline. Critically, the backbone model (ACE-Step-1.5-XL-SFT) is included as a baseline to specifically highlight the improvements gained by adding the audio-conditioned templates. The evaluation metrics are appropriate and cover both control adherence and general music quality. For control adherence, metrics like Beat-F1, Cemgil (beats), V-MSE, PER (vocals), A-MSE (accompaniment), Pitch_50 (prosody), and MuLan-A (reference) are used. The results clearly show that DiffSynth-Music significantly outperforms all baselines across all seven control adherence metrics, often by a large margin (e.g., Beat-F1 from ~0.3 to ~0.85, Pitch_50 from ~0.04 to ~0.47). This strongly validates the effectiveness of the proposed conditioning framework. To assess general music quality, AudioBox-Aesthetics scores (CE, CU, PC, PQ) and MuLan-T (text-music alignment) are used. The results indicate that DiffSynth-Music maintains "broadly comparable" scores to the evaluated base models, with some metric-specific trade-offs. While some scores are slightly lower than the backbone, the overall message is that the added controllability does not severely degrade the general generation quality, which is a reasonable and expected outcome for such an adapter-based system. The experimental setup and results convincingly support the paper's claims.
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.
Primary: StepFun
All Institutions: StepFun, ACE, The Chinese University of Hong Kong, University of California San Diego
StepAudio 3 Music introduces a robust framework for long-form music generation that integrates explicit symbolic planning (ABC-CoT) with a discrete-continuous audio pipeline, achieving competitive performance with leading commercial systems. The paper provides a thorough analysis of the trade-offs in audio tokenization for autoregressive models and demonstrates the efficacy of preference optimization in enhancing musical quality and condition adherence.
The paper proposes a hierarchical architecture for long-form music generation combining a Mixture-of-Experts (MoE) autoregressive language model with a flow-matching Diffusion Transformer (DiT) renderer. A key methodological contribution is the "ABC-CoT" (ABC Chain-of-Thought) mechanism, where the LLM generates an intermediate symbolic arrangement plan in ABC notation before predicting discrete audio tokens. This allows for explicit control over harmony, rhythm, and structure. The audio representation uses a novel 50-Hz single-codebook tokenizer (65,536 entries) trained with semantically informed self-supervised objectives, which the authors argue offers a better trade-off between acoustic fidelity and autoregressive predictability compared to multi-codebook RVQ approaches. The training pipeline is extensive, involving a progressive curriculum from large-scale pre-training to high-quality annealing, followed by SFT and DPO for preference alignment.
The evaluation is comprehensive, utilizing both objective metrics (SongBench, AudioBox-Aesthetics, MuQ-MuLan) and subjective human evaluation via the Artificial Analysis Music Arena. The model achieves top-tier results, ranking 4th in the Arena (Elo 1105) behind only Suno V5.5 and Mureka, and leading in several objective quality metrics. Ablation studies effectively demonstrate the contribution of the DPO stage and the ABC-CoT planning mechanism, showing consistent improvements in musicality and condition adherence. The comparison with commercial systems (Suno, MiniMax) provides strong external validation of the model's competitive standing.
The paper provides detailed architectural specifications, training hyperparameters, and data pipeline descriptions. However, as is common with large-scale industrial models, the specific codebase and full training data are not released. The reliance on proprietary components (StepAudio VAE, specific LLM backbone) limits full independent reproduction, though the methodology is described with sufficient clarity for understanding the design choices.
The primary limitation is the lack of open-source code and weights, which restricts the community's ability to verify results or build upon the specific implementation. The evaluation, while strong, relies heavily on the model's performance in a specific arena and a set of internal benchmarks; broader generalization to diverse musical styles or non-vocal genres is less thoroughly explored in the text. The "ABC-CoT" approach adds computational overhead during inference due to the two-pass generation process.
This work significantly advances the state of the art in controllable music generation by integrating symbolic reasoning (ABC notation) with neural audio synthesis. It demonstrates that explicit intermediate planning can improve the coherence and controllability of long-form music generation. The findings regarding the trade-offs between single-codebook VQ and RVQ for autoregressive music models provide valuable insights for future audio codec design. The high performance on human preference leaderboards indicates practical utility for creative applications. StepAudio 3 Music introduces a robust framework for long-form music generation that integrates explicit symbolic planning (ABC-CoT) with a discrete-continuous audio pipeline, achieving competitive performance with leading commercial systems. The paper provides a thorough analysis of the trade-offs in audio tokenization for autoregressive models and demonstrates the efficacy of preference optimization in enhancing musical quality and condition adherence.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Primary: Mila-Quebec AI Institute
All Institutions: Concordia University, Mila-Quebec AI Institute, Université Laval
ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
The paper proposes ZipCodec, a streaming neural speech codec that significantly reduces the frame rate to 6.25 Hz (160 ms per token) while maintaining a low bitrate of 0.80 kbps. The methodology is built upon the FocalCodec-Stream architecture but introduces several key modifications: (1) replacing the learned convolutional encoder with a causal log-mel frontend to simplify the pipeline; (2) introducing "ErfFormer," a transformer variant that removes RMSNorm and positional encodings, relying on DynamicErf activation and causal attention to handle long streaming contexts without positional extrapolation; (3) employing Scalar Spherical Quantization (SSQ) with a factorized bottleneck (64 dimensions, 4 levels) to achieve compact representation; and (4) a latency-aware streaming decoder that processes 8 WavLM representations jointly within the 160 ms window to relax causality constraints at the feature level. The training strategy involves large-scale distillation of WavLM layer-6 representations using ~94,000 hours of speech data, with specific noise and overlapping speech augmentations to match the teacher model's distribution.
The experimental evaluation is comprehensive, covering speech resynthesis (English and multilingual), voice conversion, and a suite of downstream discriminative and generative tasks (ASR, SI, SER, KS, IC, SE, SS). ZipCodec demonstrates superior performance compared to strong baselines like Mimi, PAST, and FocalCodec-Stream, particularly in intelligibility (dWER) and speaker similarity (Sim) at the significantly lower frame rate. The paper also provides detailed streaming efficiency metrics, showing real-time inference on consumer-grade CPUs and efficient GPU scaling. The results convincingly show that the 6.25 Hz bottleneck retains sufficient information for high-quality reconstruction and downstream tasks, narrowing the gap with non-streaming offline codecs.
The paper provides high reproducibility. It explicitly states that code and checkpoints are available. Detailed hyperparameters, training data sources (LibriLight, VoxPopuli, GigaSpeech), augmentation strategies, and architectural specifications (e.g., ErfFormer dimensions, SSQ parameters) are provided. The use of standard datasets and metrics (UTMOS, dWER, DNSMOS) allows for easy verification and comparison with other works in the field.
The primary limitation is the model size (842M parameters), which, while manageable for single-stream CPU inference, may be prohibitive for edge devices or high-concurrency server deployments compared to smaller codecs. Additionally, the evaluation is primarily focused on English and specific multilingual subsets; broader language coverage is not extensively tested. The reliance on WavLM distillation ties the codec's performance to the quality and biases of the WavLM teacher model.
This work has significant implications for speech-language modeling and real-time speech-to-speech systems. By reducing the frame rate to 6.25 Hz, the token sequence length is drastically shortened, which reduces computational costs and simplifies autoregressive modeling for speech LLMs. The low latency (160 ms) makes it suitable for highly responsive conversational AI applications. The open-sourcing of code and demos facilitates further research and adoption in the community. ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Primary: Imperial College London
All Institutions: Imperial College London, Technische Universität München, Mohamed bin Zayed University of Artificial Intelligence, Shanghai Jiao Tong University, Johns Hopkins University
The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
The paper proposes ARIA-Rubrics, a framework to evaluate the reasoning process of Large Audio Language Models (LALMs) rather than just final answer accuracy. The methodology relies on Chain-of-Thought (CoT) prompting to externalize reasoning into four steps: Perception, Analysis, Reasoning, and Answer. It introduces six metrics: Acoustic Grounding Score (using CLAP embeddings to verify entities against audio), Inter-step Coherence (using MPNet embeddings), Reasoning Chain Coverage (using a lightweight LLM judge for substantiveness), Reasoning Step Causality (using perplexity reduction), Reasoning Progress Score (using NLI entailment), and Audio Lexical Density (using Word2Vec similarity to AudioSet ontology). The approach is innovative in combining audio-specific perceptual checks with text-based reasoning quality metrics, addressing the gap where text-only metrics fail to detect perceptual hallucinations.
The authors evaluate 9 models across two benchmarks (MMAR and MMAU-mini). They identify three distinct reasoning modes: Reasoning, Scaffolding, and Decoration. The paper claims high correlation with human judgments, validating the automated metrics. The experimental setup is robust, covering both open-source and closed-source models, and the identification of specific failure modes (like scaffolding) provides actionable insights for model developers.
The code is available on GitHub. The paper specifies the models used for each metric (CLAP, MPNet, Qwen2.5-1.5B, RoBERTa-large-mnli, Word2Vec) and provides details on the CoT prompt structure and the AudioSet ontology filtering. This level of detail supports reproducibility, although the specific few-shot prompts for the LLM judge are referenced in appendices not fully visible in the truncated text, which is standard practice.
The framework relies on the quality of the CoT output; if a model refuses to generate CoT or generates unstructured text, the metrics may fail or be biased. The use of a small LLM (Qwen2.5-1.5B) for judging substantiveness and causality may limit the depth of evaluation compared to larger models. Additionally, the Audio Lexical Density metric is a proxy and may not capture all aspects of audio-relevant reasoning.
This work is significant for the development of reliable audio AI systems. By distinguishing between genuine reasoning and pattern matching, it provides a diagnostic tool for improving LALMs. It encourages the development of models that are not only accurate but also faithful in their reasoning, which is crucial for safety and interpretability in audio applications. The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
We present UniStream, a fully causal 48 kHz neural audio codec for streaming speech, music, and environmental sounds. At its core is Multi-Expert Residual Vector Quantization (ME-RVQ), which replaces the single shared codebook in each residual quantization layer with four expert codebooks controlled by a deterministic Top-K router. Because routing decisions are derived solely from previously decoded quantized states, the decoder can reproduce the selected experts without transmitting expert identifiers, thereby expanding quantization capacity while adding 5.5M parameters. We further introduce an auxiliary Optimal Transport Conditional Flow Matching (OT-CFM) objective to regularize the quantized latent space during training. The flow module is removed entirely at inference and therefore incurs no runtime overhead. UniStream supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode within a causal 48 kHz encoder-decoder framework, while achieving real-time GPU inference. To complement narrow-band speech metrics, we report 48 kHz ViSQOL audio mode, ViSQOL speech mode, standard VGGish-FAD, DNSMOS P.835, and higher-rate reference comparisons with Opus and EnCodec. At 12 kbps, UniStream-Top1 achieves PESQ and UTMOS scores comparable to EnCodec while reducing speech Mel-D from 13.07 to 8.21. At 22.5 kbps, UniStream-Top2 achieves a ViSQOL speech-mode score of 4.67 and an environmental audio-mode score of 3.96, exceeding all evaluated systems operating at 12 kbps or below in the latter setting. It also comes within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode. Ablation studies confirm that ME-RVQ is the primary source of quality improvement, whereas OT-CFM provides perceptual gains on speech with a mild trade-off in spectral distortion.
Primary: Tsinghua University
All Institutions: Tsinghua Shenzhen International Graduate School, Tsinghua University
UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
The paper proposes UniStream, a causal neural audio codec featuring Multi-Expert Residual Vector Quantization (ME-RVQ). The core innovation is replacing single codebooks in residual layers with four expert codebooks routed by a deterministic Top-K mechanism. Crucially, the routing is derived from previously decoded states, allowing the decoder to replicate expert selection without transmitting expert IDs, thus saving bandwidth. The method also introduces a training-only Optimal Transport Conditional Flow Matching (OT-CFM) regularizer to improve latent space quality without inference cost. The architecture is fully causal, suitable for streaming, and supports 12 kbps and 22.5 kbps modes.
Experiments are conducted on speech (LibriSpeech), music (MTG-Jamendo), and environmental audio (FSD50K). The evaluation is comprehensive, using both narrow-band metrics (PESQ, STOI, UTMOS) and full-band perceptual metrics (ViSQOL, VGGish-FAD, DNSMOS). Results show UniStream-Top1 outperforms EnCodec in spectral distortion (Mel-D) at 12 kbps, while UniStream-Top2 approaches Opus 24 kbps quality at a lower bitrate. Ablations confirm ME-RVQ is the primary driver of improvement, with OT-CFM providing modest perceptual gains.
The paper provides detailed architectural specifications (strides, dilation factors, codebook sizes, loss weights) and training hyperparameters. However, no code repository or demo link is provided in the text. The use of standard datasets and metrics facilitates reproduction, but the lack of open-source code limits immediate verification.
The paper lacks subjective listening tests (MOS), relying solely on objective metrics. Music domain performance (FAD) is noted as weaker compared to speech and environmental audio. The CPU inference time (RTF 1.05) is borderline for real-time deployment on standard hardware, though GPU inference is efficient. The comparison with non-causal baselines like EnCodec is somewhat unfair for streaming contexts, though acknowledged.
This work contributes to the field of low-latency neural audio coding, particularly for applications requiring full-band (48 kHz) fidelity in streaming scenarios. The deterministic routing mechanism offers a practical solution for expanding quantization capacity without signaling overhead, which could influence future codec designs. UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
Primary: KAIST
All Institutions: KAIST
The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
The paper proposes AV-STE, a modular front-end for enhancing semantic speech tokens in full-duplex dialogue systems. The core contribution is the decoupling of robustness enhancement from the large language model (LLM) backbone. By keeping the speech LLM (Moshi) frozen and only enhancing the semantic token stream using a streaming audio-visual encoder (modified AV-HuBERT) and a noise-adaptive fusion module, the authors address a significant practical limitation of existing multimodal dialogue systems that require costly retraining. The introduction of "Soft Token Cross-Attention" to handle uncertainty in noisy tokenizer outputs and "Noise-Adaptive Modulation" based on entropy are technically sound and well-motivated innovations. The adaptation of bidirectional AV-HuBERT to a causal, streaming architecture with bounded lookahead is a necessary and well-executed engineering contribution for real-time applications.
The experimental setup is rigorous, covering in-domain (LRS3) and out-of-domain (Seamless Interaction) datasets, with various noise types (non-speech, speaker interference). The evaluation metrics are appropriate for the task: semantic token accuracy for the front-end and GPT-4o-judged coherence plus turn-taking metrics (TOR, latency) for the downstream dialogue. The results show substantial improvements in token recovery (e.g., from 12.61% to 65.60% under same-dataset interference) and downstream coherence (1.42 to 1.91). The comparison against a non-streaming baseline (PASE) and a different dialogue model (PersonaPlex) provides useful context, although the primary comparison is against the frozen Moshi baseline. The qualitative examples strongly support the quantitative findings.
The paper provides a GitHub link to the code and models. The training configuration, dataset preprocessing, and model architecture details are described in sufficient detail for reproduction. The use of standard datasets (LRS3, AudioSet, Seamless Interaction) and open-source components (Moshi, AV-HuBERT) further enhances reproducibility.
The paper acknowledges that AV-STE assumes reliable visual input and does not handle visual occlusion or missing video. There is a noted domain shift issue where clean-token accuracy drops on out-of-domain data (Seamless Interaction), indicating sensitivity to domain characteristics. The reliance on GPT-4o for evaluation, while common, introduces a potential bias, though it is a standard practice in current LLM evaluation. The method is specific to the Moshi/Mimi tokenization scheme, limiting immediate applicability to other speech LLM architectures without adaptation.
This work has significant implications for the deployment of robust spoken dialogue systems in real-world, noisy environments. By providing a plug-and-play module that enhances robustness without retraining the expensive LLM backbone, it lowers the barrier to entry for audio-visual dialogue systems. The focus on full-duplex interaction is timely and relevant to the next generation of conversational AI. The approach could be extended to other modalities or tokenization schemes, potentially influencing the design of future multimodal speech interfaces. The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Primary: Tencent
All Institutions: Tencent, Zhejiang University, Shanghai Jiao Tong University, The Chinese University of Hong Kong, Nanyang Technological University
Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
The paper proposes "Gander," an end-to-end Omni Interaction Agent that unifies perception, real-time interaction, and agentic capabilities. The core architectural innovation is the "Cerebellum-Brain" collaborative framework. The "Cerebellum" is a streaming Thinker-Talker model responsible for low-latency, full-duplex interaction, while the "Brain" is a training-free, plug-and-play agent (e.g., Codex, Claude Code) handling complex reasoning. The Cerebellum uses a "streaming chunk flattening" mechanism, serializing audio, video, and text into a unified token stream where the model predicts control tokens (listen/speak/interrupt) at each 1-second chunk. This allows for native interruption and proactive feedback without external VAD modules. The methodology is sound, addressing the latency-intelligence trade-off by decoupling fast interaction from slow reasoning.
The evaluation is comprehensive, covering conversational ability, omni understanding, interactive capability, and agentic intelligence. The paper reports internal human evaluations showing Gander maintains SOTA spoken dialogue naturalness while achieving competitive omni interaction performance. It demonstrates robustness in noisy environments, multi-party interactions, and backchannel communication. However, the paper relies heavily on "internal human evaluations" and subjective demos, with limited quantitative benchmarks compared to specific baselines (e.g., GPT-4o, MiniCPM-o) in the provided text snippet. The lack of standardized public benchmark scores for the "agentic" component makes direct comparison difficult.
The authors state they release the model weights, code, and data. The architecture details (chunk size, token rates, encoder types like SigLIP) are provided. However, the "Brain" component relies on proprietary external agents (Codex, Claude Code), which limits full reproducibility of the agentic performance. The training data pipeline is described in detail, including the synthesis of full-duplex interaction data, which aids reproducibility of the Cerebellum component.
The system's performance is coupled to the quality of the external "Brain" agent, which may have its own latency and cost implications. The "training-free" nature of the Brain means the Cerebellum cannot learn from the Brain's specific failures or adapt its tool-calling strategy through end-to-end fine-tuning. The evaluation lacks extensive quantitative ablation studies comparing the Cerebellum-Brain architecture against monolithic models on standardized agentic benchmarks.
This work is significant for the development of natural, human-like AI assistants that can handle complex tasks while maintaining fluid conversation. The decoupled architecture offers a scalable path for integrating powerful reasoning models into real-time interfaces. It pushes the field beyond turn-based chatbots toward continuous, multimodal collaboration. Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.