Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erberโs Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of Chinese Academy of Sciences, Shanghai Jiao Tong University
VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
The paper proposes VibeVoice-ASR-Streaming, an end-to-end LLM-based framework for streaming speaker-attributed ASR. The core methodological contribution is the adaptation of the VibeVoice-ASR architecture (which unifies ASR and diarization) to a streaming setting by interleaving fixed-size audio chunks with generated text and retaining full conversation history in the LLM context. The authors introduce a fixed lookahead mechanism (4 latent frames) to handle boundary effects. The training strategy involves a three-stage process: non-streaming pre-training, streaming pre-training with chunked data, and streaming fine-tuning. The approach relies on the Qwen2.5 LLM backbone and dual tokenizers (Acoustic and Semantic) from the VibeVoice series. The methodology is sound, leveraging the strong contextual capabilities of LLMs to maintain speaker identity over long durations without explicit diarization modules, which is a significant architectural simplification compared to cascaded systems.
The experimental evaluation is comprehensive, covering four meeting benchmarks (AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM) and the multilingual MLC-Challenge benchmark. The 7B model achieves the lowest average WER/CER among compared streaming systems and the best or tied-best cpWER/cpCER in 12 of 13 settings. The paper provides detailed ablations on chunk size, model scale, speaker-label placement, and lookahead depth. Notably, it compares against both specialized streaming ASR models and cloud services (Azure, Google), demonstrating a significant latency advantage (2.00s vs 8-9s) while maintaining or improving accuracy. The analysis of the "cost of streaming conversion" (degradation from offline to streaming) is particularly insightful, quantifying the trade-off.
The paper is highly reproducible. The authors release both 1.5B and 7B model weights along with inference code. Detailed training data construction methods, including the use of Qwen3-ForcedAligner for word-level timing and specific augmentation strategies, are described. The exact hyperparameters for training stages are provided in the appendix. The evaluation protocol, including the specific versions of cloud services and normalization rules, is clearly defined.
The primary limitations are the 8-minute recording length cap due to context window/compute constraints, the limited multilingual coverage (10 languages) dictated by the forced aligner, and performance degradation on long-duration overlaps due to the serialized output format. The first-packet latency (3.5s for 22-frame chunks) is higher than the steady-state latency, which may be a concern for real-time applications.
This work has significant impact on the development of real-time voice assistants and agents. By unifying ASR and speaker diarization in a streaming LLM, it removes the need for complex cascaded pipelines, simplifying system architecture and reducing latency. The open-sourcing of the models and code will likely accelerate research in streaming speech processing and enable the deployment of more robust, multi-speaker aware voice interfaces. VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.
Primary: University of Houston
All Institutions: University of Houston, Stevens Institute of Technology, Waseda University
[One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
The paper proposes PhysWave, a physics-guided latent diffusion model for text-to-First-Order Ambisonics (FOA) generation. The core methodological contribution is the integration of differentiable acoustic priors (spherical-harmonic direction consistency and inverse-square distance consistency) into the diffusion training objective. By decoding the predicted clean latent back to the waveform during training and applying these physical constraints, the model explicitly enforces spatial accuracy rather than relying solely on implicit data-driven learning. Additionally, the paper introduces a unified waypoint-caption representation that allows both natural language descriptions and precise parametric trajectories to condition the same model, bridging the gap between usability and precision.
The authors construct a 300K-clip synthetic FOA dataset, which is a significant resource contribution. Experiments include ablations on the physics losses, comparisons with state-of-the-art methods (ImmerseDiffusion, SonicMotion), and subjective listening tests. The results show that the physics priors significantly reduce angular error and improve distance consistency without degrading audio quality metrics (FAD, CLAP). The subjective evaluation with 14 participants confirms the perceptual benefits.
The paper provides detailed implementation details, including the VAE architecture, diffusion model hyperparameters, and the specific prompts used for the LLM parser. The release of the 300K dataset and the project website enhances reproducibility. However, the reliance on a specific LLM (Qwen3.5-4B) for parsing might introduce variability, though the deterministic post-processing helps.
The model currently focuses on single-source trajectories in a free-field setting, ignoring room effects like reverberation and occlusion. It also does not handle multi-source scenes. The synthetic nature of the dataset, while large, may not fully capture the complexity of real-world spatial audio recordings.
This work has significant potential for the gaming, film, and VR industries by enabling non-experts to create high-quality spatial audio via text. The physics-guided approach could be extended to other domains where physical constraints are known, such as video generation or scientific simulation. [One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of Chinese Academy of Sciences, Shanghai Jiao Tong University
VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
The paper proposes VibeVoice-ASR-Streaming, an end-to-end LLM-based framework for streaming speaker-attributed ASR. The core methodological contribution is the adaptation of the VibeVoice-ASR architecture (which unifies ASR and diarization) to a streaming setting by interleaving fixed-size audio chunks with generated text and retaining full conversation history in the LLM context. The authors introduce a fixed lookahead mechanism (4 latent frames) to handle boundary effects. The training strategy involves a three-stage process: non-streaming pre-training, streaming pre-training with chunked data, and streaming fine-tuning. The approach relies on the Qwen2.5 LLM backbone and dual tokenizers (Acoustic and Semantic) from the VibeVoice series. The methodology is sound, leveraging the strong contextual capabilities of LLMs to maintain speaker identity over long durations without explicit diarization modules, which is a significant architectural simplification compared to cascaded systems.
The experimental evaluation is comprehensive, covering four meeting benchmarks (AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM) and the multilingual MLC-Challenge benchmark. The 7B model achieves the lowest average WER/CER among compared streaming systems and the best or tied-best cpWER/cpCER in 12 of 13 settings. The paper provides detailed ablations on chunk size, model scale, speaker-label placement, and lookahead depth. Notably, it compares against both specialized streaming ASR models and cloud services (Azure, Google), demonstrating a significant latency advantage (2.00s vs 8-9s) while maintaining or improving accuracy. The analysis of the "cost of streaming conversion" (degradation from offline to streaming) is particularly insightful, quantifying the trade-off.
The paper is highly reproducible. The authors release both 1.5B and 7B model weights along with inference code. Detailed training data construction methods, including the use of Qwen3-ForcedAligner for word-level timing and specific augmentation strategies, are described. The exact hyperparameters for training stages are provided in the appendix. The evaluation protocol, including the specific versions of cloud services and normalization rules, is clearly defined.
The primary limitations are the 8-minute recording length cap due to context window/compute constraints, the limited multilingual coverage (10 languages) dictated by the forced aligner, and performance degradation on long-duration overlaps due to the serialized output format. The first-packet latency (3.5s for 22-frame chunks) is higher than the steady-state latency, which may be a concern for real-time applications.
This work has significant impact on the development of real-time voice assistants and agents. By unifying ASR and speaker diarization in a streaming LLM, it removes the need for complex cascaded pipelines, simplifying system architecture and reducing latency. The open-sourcing of the models and code will likely accelerate research in streaming speech processing and enable the deployment of more robust, multi-speaker aware voice interfaces. VibeVoice-ASR-Streaming introduces a pioneering LLM-based end-to-end approach to streaming speaker-attributed ASR, achieving state-of-the-art performance in both transcription accuracy and speaker attribution with significantly lower latency than existing cloud services. The paper demonstrates that retaining full conversation history in the LLM context is sufficient for consistent speaker labeling, eliminating the need for separate diarization stages, and provides a rigorous analysis of the trade-offs between chunk size, lookahead, and model scale in streaming speech recognition.
In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models, without task-specific architectures.
Primary: Meta
All Institutions: Meta, Queen Mary University of London
The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
The paper proposes a novel post-training recipe for Audio Language Models (ALMs) to perform automatic music upmixing. The core method involves two stages: Rejection Sampling Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO). The key innovation is the "Sphere" reward function, a deterministic suite of six perceptually-motivated sub-rewards (centering, spread, level balance, spotlight prominence, per-role consistency) derived from music mixing conventions. The authors demonstrate that encoding expert domain knowledge as verifiable rewards allows a small student model (Qwen2.5-Omni 3B) to surpass larger frontier models (Gemini 2.5 Pro) in both objective reward scores and subjective human preference. The methodology is sound, leveraging the semantic understanding of ALMs rather than task-specific encoders, and effectively addresses the lack of paired spatial mixing data by using a teacher model to generate high-quality synthetic data filtered by the reward function.
The experimental evaluation is rigorous and comprehensive. The authors use the MedleyDB dataset, segmenting it into 30-second windows to fit context limits. They conduct a subjective A/B listening test with 44 participants, showing a 76.8% preference for high-reward mixes, validating the alignment of the Sphere reward with human perception. Objective evaluation on 618 held-out test points shows the post-trained model achieving a total reward of 4.981, significantly outperforming the zero-shot teacher (4.548) and other baselines. Ablation studies are thorough, demonstrating that data quality (stricter filtering) is more important than quantity for SFT, and that removing specific sub-rewards leads to reward hacking (e.g., collapsing spatial spread). The inclusion of a multi-stimulus preference test with expert listeners further strengthens the results, showing consistent improvements from SFT to SFT+RL.
The paper provides detailed descriptions of the training setup, including hyperparameters for LoRA, learning rates, and the RL process (GRPO with PPO-style clipping). The prompt template is fully disclosed in the appendix, which is crucial for reproducibility given the reliance on prompt engineering. However, the use of proprietary models (Gemini 2.5 Pro as teacher) and a proprietary HRTF spatializer limits full reproducibility. The code and data are not explicitly linked in the provided text, though a project demo website is mentioned. The detailed ablation on reward shaping (Gaussian vs. Sigmoid vs. Step) provides valuable insights for practitioners.
The primary limitation is the context window constraint, which limits the number of stems (max 29) and segment length (30 seconds), excluding 23.5% of the dataset. This may not capture long-term musical dynamics. Additionally, the reliance on a proprietary teacher model for data generation introduces potential biases. The subjective tests, while statistically significant, involve a relatively small number of participants (44 for A/B, 10 for multi-stimulus). The reward function, while aligned with human preference, is heuristic-based and may not capture all aspects of musical aesthetics.
This work demonstrates a practical recipe for distilling expert domain knowledge into language models via verifiable rewards, applicable beyond music mixing to other data-limited domains. It highlights the potential of ALMs for complex audio tasks that require semantic understanding, moving beyond simple feature extraction. The approach could inspire similar post-training strategies for other audio generation or processing tasks where ground truth data is scarce but expert heuristics are available. The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.
Primary: Chung-Ang University
All Institutions: Chung-Ang University
The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
The paper proposes "stride-k subsampling," a deterministic, train-free indexing operation to reduce the number of audio tokens in Whisper-based models. The method exploits the temporal redundancy inherent in Whisper's preprocessing pipeline (overlapping STFT windows and convolutional stem receptive fields). By retaining every k-th token either after the convolutional stem (input-side) or after the encoder transformer (output-side), the authors reduce sequence length without retraining. The methodology is grounded in a theoretical analysis of temporal overlap, showing that for k=2, the retained tokens still cover the full input span due to receptive field overlap. The authors further employ Centered Kernel Alignment (CKA) to diagnose why output-side subsampling is more stable than input-side, attributing it to attention-induced redistribution of redundancy in the encoder. The approach is simple, parameter-free, and easily integrable into existing pipelines.
The experimental evaluation is comprehensive, covering five Whisper scales (tiny to large-v3) and three ASR benchmarks (LibriTTS, ESD, Common Voice). The results demonstrate that k=2 preserves baseline WER on clean speech with minimal cost, while harder benchmarks show larger degradations. The paper extends the evaluation to three Whisper-based SpeechLMs (Audio Flamingo 3, Qwen2-Audio, LLaMA-Omni 2), showing modest accuracy drops on MMSU and MMAU benchmarks. The efficiency gains are significant, with 52-58% reduction in GFLOPs and 19-27% reduction in end-to-end latency. The inclusion of a noise robustness analysis (SNR sweep) and a comparison with other pruning methods (SpeechPrune) strengthens the evaluation. The "hurt/help" analysis provides insight into which types of tasks are most affected by temporal subsampling.
The paper provides detailed implementation details, including the specific subsampling positions, FLOPs calculation formulas, and dataset preparation steps. The use of standard open-source models and benchmarks (HuggingFace checkpoints, LibriTTS, Common Voice) enhances reproducibility. The authors mention using a fixed random seed and provide code snippets for the subsampling operation. However, no explicit GitHub repository link is provided in the text, which slightly limits immediate reproducibility, though the method is simple enough to implement from the description.
The primary limitation is that the method is specific to encoders with overlapping windowed frontends (like Whisper's log-mel). It does not generalize to raw-waveform encoders like wav2vec 2.0 or HuBERT, which collapse under subsampling. The method requires access to the encoder's internal computation, making it inapplicable to closed-API models. Additionally, the performance degradation on noisy or difficult benchmarks (Common Voice) is significant, limiting its utility in low-resource or noisy environments. The paper acknowledges that the redundancy exploited is partly shaped by training, and future work could explore training models to tolerate subsampling.
The paper contributes to the efficiency of speech recognition and speech language models, which are increasingly deployed in real-time applications. By reducing inference cost without retraining, the method offers a practical solution for deploying Whisper-based systems on resource-constrained devices. The insights into temporal redundancy in speech encoders may inform future architectural designs. The findings on the asymmetry between input-side and output-side subsampling provide valuable guidance for model optimization. The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
Speech enhancement models are widely deployed as frontend modules in real-time speech services, yet their vulnerability to backdoor attacks remains unexplored. Existing backdoor methods are confined to classification tasks and rely on active trigger injection, an assumption incompatible with the passive processing nature of speech enhancement models. In this paper, we propose Ouroboros, a novel backdoor attack framework that leverages the ideal clean outputs of speech enhancement models as natural triggers, enabling inference-time activation without any external trigger injection. Extensive evaluations show Ouroboros achieves near-perfect attack success rates with minimal performance degradation on diverse models and datasets. Physical-world validations confirm that naturally recorded, unaltered clean audio can reliably activate the backdoor. Moreover, Ouroboros generalizes to targeted content-tampering attacks and remains effective against common filtering and finetuning defenses.
Primary: Ningbo University
All Institutions: Ningbo University, Ningbo University of Finance and Economics
The paper introduces a novel backdoor attack for speech enhancement that uses clean audio as a natural trigger, demonstrating high effectiveness and stealthiness across multiple architectures and real-world scenarios. By shifting the trigger from artificial injections to natural high-SNR speech, the work addresses a realistic threat model for passive audio processing services, providing significant insights into the security vulnerabilities of current speech enhancement pipelines.
The paper proposes "Ouroboros," a backdoor attack framework for speech enhancement models that utilizes clean audio as the trigger. The core innovation is the "CleanTrigger" mechanism, which repurposes the model's expected output (clean speech) as the input trigger. This is a clever conceptual shift from traditional backdoor attacks that require active injection of artificial signals (e.g., ultrasonic tones or specific noise patterns). By targeting high-SNR samples in the training data and poisoning them to map to a malicious target (silence or a specific phrase), the attack leverages the natural occurrence of clean audio in real-world scenarios (e.g., a user speaking clearly in a quiet room). The methodology is sound, utilizing an SNR-based selection strategy to minimize degradation on the primary task (denoising noisy speech) while maximizing attack success on clean speech. The threat model is realistic for supply-chain attacks where an attacker poisons the training dataset without needing to manipulate inference-time inputs.
The experiments are comprehensive, covering four distinct speech enhancement architectures (MP-SENet, SEMamba, CMGAN, FlowSE) and two standard datasets (VoiceBank-Demand, WSJ0-CHiME3). The results demonstrate near-perfect Attack Success Rates (ASR) with minimal degradation in PESQ scores compared to the BadNets baseline. The inclusion of physical-world validation (recording 60 clean voices via smartphone) is a strong point, confirming the practical feasibility of the attack. The extension to content-tampering (outputting a specific phrase) further validates the versatility of the attack. The ablation studies on poisoning rates and SNR selection strategies provide good insight into the trade-offs.
The paper provides sufficient details for reproduction, including the specific SNR thresholds, poisoning rates, and model architectures used. However, the lack of a public code repository (no GitHub link provided in the text) slightly hinders immediate reproducibility. The use of standard datasets and open-source models (like CMGAN and FlowSE) mitigates this to some extent.
The primary limitation is the assumption that the attacker has access to poison the training data, which is a strong assumption for many deployed systems. Additionally, the attack relies on the model being trained on paired data; it may not generalize directly to unpaired or self-supervised enhancement models. The "silence" attack is a disruption attack, and while the content-tampering extension is shown, the robustness of the specific phrase output against different TTS voices or languages is not deeply explored.
This work highlights a critical security vulnerability in passive audio processing pipelines. As speech enhancement becomes ubiquitous in real-time communication and voice assistants, the ability to silently disable or manipulate these services using natural audio inputs poses a significant risk. The findings urge the community to consider backdoor robustness in the design of speech enhancement models, particularly those deployed in safety-critical or privacy-sensitive applications. The paper introduces a novel backdoor attack for speech enhancement that uses clean audio as a natural trigger, demonstrating high effectiveness and stealthiness across multiple architectures and real-world scenarios. By shifting the trigger from artificial injections to natural high-SNR speech, the work addresses a realistic threat model for passive audio processing services, providing significant insights into the security vulnerabilities of current speech enhancement pipelines.
Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ฯ=-0.068$, PESQ $ฯ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.
Primary: GN Group
All Institutions: GN Group
The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
The paper introduces Output Divergence Rate (ODR), a metric designed to quantify the semantic impact of speech enhancement (SE) on downstream Large Language Model (LLM) intent classification. The methodology is sound, utilizing a cascaded pipeline of ASR (Whisper large-v3 and wav2vec2-large) followed by LLM classification (Gemini 2.5 Flash Lite). The approach correctly identifies a critical gap in current evaluation practices: standard audio quality metrics (PESQ, STOI) and transcription metrics (WER) do not correlate well with semantic task performance. The use of a closed-set intent classification task (SLURP) provides a controlled environment to measure divergence, although it limits the scope to specific command structures. The decomposition of divergence into "corrections" vs. "regressions" using ground truth labels is a rigorous addition that validates the metric's sensitivity to actual harm rather than random noise.
The experiments are comprehensive, covering five distinct enhancement conditions (Noisy, MetricGAN+, Echo, Echo+AEC, Dereverb) across 2,974 clips. The results are striking: MetricGAN+ doubles the ODR compared to unenhanced noisy speech despite improving PESQ, and unmitigated echo causes catastrophic failure (ODR 0.836) due to speaker substitution, a failure mode WER cannot capture. The replication across two architecturally distinct ASR models (attention-based vs. CTC) strengthens the findings, showing that the issue is not model-specific. The analysis of within-condition vs. pooled correlations is particularly valuable, demonstrating that while metrics separate conditions, they fail to predict per-clip divergence, which is crucial for deployment monitoring.
High. The authors release the full pipeline on GitHub. The use of standard, publicly available datasets (SLURP, DNS Challenge) and models (Whisper, MetricGAN+, Gemini API) ensures that the results are reproducible by other researchers. The prompt used for LLM classification is provided in the appendix, and hyperparameters for ASR and SE models are specified.
The study relies on simulated degradations (DNS Challenge) rather than real-world field recordings, which may differ in acoustic complexity. The LLM evaluation is limited to a single model family (Gemini) for the primary results, with only partial replication on a larger model (Gemini Pro). The task is restricted to English and closed-set intent classification, so findings may not generalize to open-ended generation or other languages. ODR requires a clean reference, limiting its use as an online monitor.
This paper has significant implications for the deployment of spoken AI systems. It challenges the common assumption that improving audio quality metrics (like PESQ) leads to better user experience in LLM-based assistants. It provides a new diagnostic tool (ODR) for developers to evaluate SE front-ends specifically for their impact on semantic understanding. The finding that standard metrics are poor predictors of LLM failure urges the field to adopt task-specific evaluation protocols for voice interfaces. The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.
Primary: University of Houston
All Institutions: University of Houston, Stevens Institute of Technology, Waseda University
[One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
The paper proposes PhysWave, a physics-guided latent diffusion model for text-to-First-Order Ambisonics (FOA) generation. The core methodological contribution is the integration of differentiable acoustic priors (spherical-harmonic direction consistency and inverse-square distance consistency) into the diffusion training objective. By decoding the predicted clean latent back to the waveform during training and applying these physical constraints, the model explicitly enforces spatial accuracy rather than relying solely on implicit data-driven learning. Additionally, the paper introduces a unified waypoint-caption representation that allows both natural language descriptions and precise parametric trajectories to condition the same model, bridging the gap between usability and precision.
The authors construct a 300K-clip synthetic FOA dataset, which is a significant resource contribution. Experiments include ablations on the physics losses, comparisons with state-of-the-art methods (ImmerseDiffusion, SonicMotion), and subjective listening tests. The results show that the physics priors significantly reduce angular error and improve distance consistency without degrading audio quality metrics (FAD, CLAP). The subjective evaluation with 14 participants confirms the perceptual benefits.
The paper provides detailed implementation details, including the VAE architecture, diffusion model hyperparameters, and the specific prompts used for the LLM parser. The release of the 300K dataset and the project website enhances reproducibility. However, the reliance on a specific LLM (Qwen3.5-4B) for parsing might introduce variability, though the deterministic post-processing helps.
The model currently focuses on single-source trajectories in a free-field setting, ignoring room effects like reverberation and occlusion. It also does not handle multi-source scenes. The synthetic nature of the dataset, while large, may not fully capture the complexity of real-world spatial audio recordings.
This work has significant potential for the gaming, film, and VR industries by enabling non-experts to create high-quality spatial audio via text. The physics-guided approach could be extended to other domains where physical constraints are known, such as video generation or scientific simulation. [One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erberโs Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.
Primary: National University of Singapore
All Institutions: National University of Singapore, Tencent, LIGHTSPEED, Nanyang Technological University, Independent Researcher, The Chinese University of Hong Kong, Shenzhen, Shenzhen Loop Area Institute
The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
The paper proposes "LoopTTS," a closed-loop TTS framework comprising a Filter, Judge (AudioLLM), and Refiner. The core methodological contribution is the Refiner, a fine-tuned TTS model (based on EmoVoice) that performs guided expressive re-synthesis conditioned on the initial defective audio, target text, and structured instructions from the AudioLLM Judge. The training methodology introduces a "Position-Weighted Cross-Entropy Loss" and a "Structural Alignment Loss" using Representational Similarity Matrices (RSMs) to focus supervision on sparse prosodic spans (stress/pause). The data construction pipeline ("Refiner-DB") uses contrastive AudioLLM annotation to generate weak supervision for word-level prosody. The approach is technically sound, addressing the specific gap of local prosodic defects that utterance-level metrics miss. The use of RSMs for span-level consistency is a creative application of representation learning to TTS fine-tuning.
The experiments are rigorous and well-structured. The authors evaluate the full pipeline on a subset of flagged utterances (200 samples) and a larger scaled evaluation (3,000 samples). They include strong baselines: open-loop re-generation (CosyVoice2, EmoVoice), global-only control, and a human-instruction upper bound. The use of a different AudioLLM (Qwen3-Omni) for the Judge stage demonstrates robustness. Subjective evaluations (MOS, MOS-I) are conducted by professional evaluators with reported ICC reliability. The results show consistent improvements in recovery quality and instruction-following fidelity (stress/pause) over baselines. The ablation studies effectively isolate the contributions of the position-weighting and structural alignment losses.
The paper provides high reproducibility. Code, data construction scripts, prompts, and model checkpoints are promised for release. The GitHub link is provided. Detailed hyperparameters, loss formulations, and dataset composition are included in the main text and appendices. The use of public datasets (ESD, RAVDESS, etc.) and standard models (CosyVoice2, Whisper) further enhances reproducibility.
The primary limitation is the reliance on a proprietary AudioLLM (Gemini-3-pro) for both data annotation and inference-time judging, which may limit accessibility and introduce potential biases. The evaluation is focused on English speech, and the framework's applicability to tonal languages is not tested. The "weak supervision" from AudioLLMs is acknowledged as imperfect, with moderate F1 scores for pause detection. The system is designed for offline batch processing, not streaming, due to the multi-stage latency.
This work has significant implications for the deployment of high-quality TTS systems in production environments where sporadic prosodic errors are unacceptable. It demonstrates a viable path for integrating LLM-based diagnostic capabilities into generative audio pipelines, moving beyond simple generation to generation-and-correction. The methodology of using AudioLLMs for scalable weak supervision of fine-grained prosodic features could be extended to other audio tasks, such as music production or sound effect design. The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.