Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Amazon AGI Foundations
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
The paper introduces SpeechGym, a novel audio-native agentic environment designed for training voice agents via reinforcement learning. The core methodological contribution is the creation of an end-to-end trainable loop for multi-turn dialogue and tool use in speech, which is claimed to be a first. The environment integrates two omni-modal models (Qwen3-Omni-30B-A3B) – a frozen user model and a trainable agent model – allowing for native audio interaction without external ASR/TTS or proprietary APIs. This local setup is crucial for enabling gradient flow and affordable on-policy RL. The problem is formulated as a POMDP, where the agent decides between structured tool calls (textual) and variable-length speech responses. Keeping tool calls textual is a sensible design choice, isolating perceptual errors from reasoning errors. The environment inherits tasks, tools, databases, and success checks from the `tau2` text benchmark, ensuring modality is the primary variable. A new "Banking" domain is introduced, specifically stressing high-stakes numeric slots, which is a valuable addition for evaluating robustness to perceptual errors. The reward mechanism is based on `tau2`'s outcome check, a product of binary components, ensuring strict task completion. To address the severe sparsity of this reward in audio settings, the paper proposes a per-turn process shaping, crediting successful tool executions and penalizing failures. This is a classical reward shaping technique adapted effectively to the agentic setting. For efficient rollout collection, the authors leverage `vLLM-Omni`, serving both models and the LoRA adapter, which significantly speeds up training epochs and eliminates API costs, making online speech RL practical. The overall methodology is well-conceived, addressing key challenges in audio agent training.
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
AudioSpan introduces a meticulously designed benchmark for long-form audio comprehension, addressing critical gaps in existing LALM evaluation. The core methodology revolves around two complementary question paths: Native QA and Anchor QA, both built upon a three-level cognitive taxonomy (Perception, Understanding, Reasoning). Native QA derives questions from the audio's inherent content, offering both multiple-choice (MCQ) and open-ended (OEQ) formats, with OEQs graded by an LLM ensemble using detailed rubrics that include semantic and temporal scores. Anchor QA is particularly innovative, injecting acoustic anchors into the audio to create verifiable ground truth and constructing a perception-to-reasoning chain of MCQs, scored only up to the first error to penalize shortcutting. This dual-path approach effectively combines broad coverage of natural content with precisely verifiable ground truth. The fully automated construction pipeline is a significant strength. Phase 1 generates a structured caption from the audio, involving ASR, semantic segmentation, acoustic captioning, dependency analysis, and caption assembly into evidence records. Phase 2 then generates QA pairs, with a sophisticated multi-level quality assurance system. This system includes evidence verification against the audio, a three-level solver (blind, text-only, full access) to filter out guessable or text-only answerable items, and a novel adversarial critic feedback loop that refines question generation by learning from past failure modes. This adaptive refinement mechanism is a key technical contribution, ensuring high-quality, audio-dependent questions at scale. The detailed rubric for OEQs, incorporating temporal grounding as a critical component, is also well-conceived and crucial for evaluating true comprehension.
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erber’s Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models, without task-specific architectures.
Primary: Meta
All Institutions: Meta, Queen Mary University of London
The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
The paper proposes a novel post-training recipe for Audio Language Models (ALMs) to perform automatic music upmixing. The core method involves two stages: Rejection Sampling Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO). The key innovation is the "Sphere" reward function, a deterministic suite of six perceptually-motivated sub-rewards (centering, spread, level balance, spotlight prominence, per-role consistency) derived from music mixing conventions. The authors demonstrate that encoding expert domain knowledge as verifiable rewards allows a small student model (Qwen2.5-Omni 3B) to surpass larger frontier models (Gemini 2.5 Pro) in both objective reward scores and subjective human preference. The methodology is sound, leveraging the semantic understanding of ALMs rather than task-specific encoders, and effectively addresses the lack of paired spatial mixing data by using a teacher model to generate high-quality synthetic data filtered by the reward function.
The experimental evaluation is rigorous and comprehensive. The authors use the MedleyDB dataset, segmenting it into 30-second windows to fit context limits. They conduct a subjective A/B listening test with 44 participants, showing a 76.8% preference for high-reward mixes, validating the alignment of the Sphere reward with human perception. Objective evaluation on 618 held-out test points shows the post-trained model achieving a total reward of 4.981, significantly outperforming the zero-shot teacher (4.548) and other baselines. Ablation studies are thorough, demonstrating that data quality (stricter filtering) is more important than quantity for SFT, and that removing specific sub-rewards leads to reward hacking (e.g., collapsing spatial spread). The inclusion of a multi-stimulus preference test with expert listeners further strengthens the results, showing consistent improvements from SFT to SFT+RL.
The paper provides detailed descriptions of the training setup, including hyperparameters for LoRA, learning rates, and the RL process (GRPO with PPO-style clipping). The prompt template is fully disclosed in the appendix, which is crucial for reproducibility given the reliance on prompt engineering. However, the use of proprietary models (Gemini 2.5 Pro as teacher) and a proprietary HRTF spatializer limits full reproducibility. The code and data are not explicitly linked in the provided text, though a project demo website is mentioned. The detailed ablation on reward shaping (Gaussian vs. Sigmoid vs. Step) provides valuable insights for practitioners.
The primary limitation is the context window constraint, which limits the number of stems (max 29) and segment length (30 seconds), excluding 23.5% of the dataset. This may not capture long-term musical dynamics. Additionally, the reliance on a proprietary teacher model for data generation introduces potential biases. The subjective tests, while statistically significant, involve a relatively small number of participants (44 for A/B, 10 for multi-stimulus). The reward function, while aligned with human preference, is heuristic-based and may not capture all aspects of musical aesthetics.
This work demonstrates a practical recipe for distilling expert domain knowledge into language models via verifiable rewards, applicable beyond music mixing to other data-limited domains. It highlights the potential of ALMs for complex audio tasks that require semantic understanding, moving beyond simple feature extraction. The approach could inspire similar post-training strategies for other audio generation or processing tasks where ground truth data is scarce but expert heuristics are available. The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.
Primary: Chung-Ang University
All Institutions: Chung-Ang University
The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
The paper proposes "stride-k subsampling," a deterministic, train-free indexing operation to reduce the number of audio tokens in Whisper-based models. The method exploits the temporal redundancy inherent in Whisper's preprocessing pipeline (overlapping STFT windows and convolutional stem receptive fields). By retaining every k-th token either after the convolutional stem (input-side) or after the encoder transformer (output-side), the authors reduce sequence length without retraining. The methodology is grounded in a theoretical analysis of temporal overlap, showing that for k=2, the retained tokens still cover the full input span due to receptive field overlap. The authors further employ Centered Kernel Alignment (CKA) to diagnose why output-side subsampling is more stable than input-side, attributing it to attention-induced redistribution of redundancy in the encoder. The approach is simple, parameter-free, and easily integrable into existing pipelines.
The experimental evaluation is comprehensive, covering five Whisper scales (tiny to large-v3) and three ASR benchmarks (LibriTTS, ESD, Common Voice). The results demonstrate that k=2 preserves baseline WER on clean speech with minimal cost, while harder benchmarks show larger degradations. The paper extends the evaluation to three Whisper-based SpeechLMs (Audio Flamingo 3, Qwen2-Audio, LLaMA-Omni 2), showing modest accuracy drops on MMSU and MMAU benchmarks. The efficiency gains are significant, with 52-58% reduction in GFLOPs and 19-27% reduction in end-to-end latency. The inclusion of a noise robustness analysis (SNR sweep) and a comparison with other pruning methods (SpeechPrune) strengthens the evaluation. The "hurt/help" analysis provides insight into which types of tasks are most affected by temporal subsampling.
The paper provides detailed implementation details, including the specific subsampling positions, FLOPs calculation formulas, and dataset preparation steps. The use of standard open-source models and benchmarks (HuggingFace checkpoints, LibriTTS, Common Voice) enhances reproducibility. The authors mention using a fixed random seed and provide code snippets for the subsampling operation. However, no explicit GitHub repository link is provided in the text, which slightly limits immediate reproducibility, though the method is simple enough to implement from the description.
The primary limitation is that the method is specific to encoders with overlapping windowed frontends (like Whisper's log-mel). It does not generalize to raw-waveform encoders like wav2vec 2.0 or HuBERT, which collapse under subsampling. The method requires access to the encoder's internal computation, making it inapplicable to closed-API models. Additionally, the performance degradation on noisy or difficult benchmarks (Common Voice) is significant, limiting its utility in low-resource or noisy environments. The paper acknowledges that the redundancy exploited is partly shaped by training, and future work could explore training models to tolerate subsampling.
The paper contributes to the efficiency of speech recognition and speech language models, which are increasingly deployed in real-time applications. By reducing inference cost without retraining, the method offers a practical solution for deploying Whisper-based systems on resource-constrained devices. The insights into temporal redundancy in speech encoders may inform future architectural designs. The findings on the asymmetry between input-side and output-side subsampling provide valuable guidance for model optimization. The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
Speech enhancement models are widely deployed as frontend modules in real-time speech services, yet their vulnerability to backdoor attacks remains unexplored. Existing backdoor methods are confined to classification tasks and rely on active trigger injection, an assumption incompatible with the passive processing nature of speech enhancement models. In this paper, we propose Ouroboros, a novel backdoor attack framework that leverages the ideal clean outputs of speech enhancement models as natural triggers, enabling inference-time activation without any external trigger injection. Extensive evaluations show Ouroboros achieves near-perfect attack success rates with minimal performance degradation on diverse models and datasets. Physical-world validations confirm that naturally recorded, unaltered clean audio can reliably activate the backdoor. Moreover, Ouroboros generalizes to targeted content-tampering attacks and remains effective against common filtering and finetuning defenses.
Primary: Ningbo University
All Institutions: Ningbo University, Ningbo University of Finance and Economics
The paper introduces a novel backdoor attack for speech enhancement that uses clean audio as a natural trigger, demonstrating high effectiveness and stealthiness across multiple architectures and real-world scenarios. By shifting the trigger from artificial injections to natural high-SNR speech, the work addresses a realistic threat model for passive audio processing services, providing significant insights into the security vulnerabilities of current speech enhancement pipelines.
The paper proposes "Ouroboros," a backdoor attack framework for speech enhancement models that utilizes clean audio as the trigger. The core innovation is the "CleanTrigger" mechanism, which repurposes the model's expected output (clean speech) as the input trigger. This is a clever conceptual shift from traditional backdoor attacks that require active injection of artificial signals (e.g., ultrasonic tones or specific noise patterns). By targeting high-SNR samples in the training data and poisoning them to map to a malicious target (silence or a specific phrase), the attack leverages the natural occurrence of clean audio in real-world scenarios (e.g., a user speaking clearly in a quiet room). The methodology is sound, utilizing an SNR-based selection strategy to minimize degradation on the primary task (denoising noisy speech) while maximizing attack success on clean speech. The threat model is realistic for supply-chain attacks where an attacker poisons the training dataset without needing to manipulate inference-time inputs.
The experiments are comprehensive, covering four distinct speech enhancement architectures (MP-SENet, SEMamba, CMGAN, FlowSE) and two standard datasets (VoiceBank-Demand, WSJ0-CHiME3). The results demonstrate near-perfect Attack Success Rates (ASR) with minimal degradation in PESQ scores compared to the BadNets baseline. The inclusion of physical-world validation (recording 60 clean voices via smartphone) is a strong point, confirming the practical feasibility of the attack. The extension to content-tampering (outputting a specific phrase) further validates the versatility of the attack. The ablation studies on poisoning rates and SNR selection strategies provide good insight into the trade-offs.
The paper provides sufficient details for reproduction, including the specific SNR thresholds, poisoning rates, and model architectures used. However, the lack of a public code repository (no GitHub link provided in the text) slightly hinders immediate reproducibility. The use of standard datasets and open-source models (like CMGAN and FlowSE) mitigates this to some extent.
The primary limitation is the assumption that the attacker has access to poison the training data, which is a strong assumption for many deployed systems. Additionally, the attack relies on the model being trained on paired data; it may not generalize directly to unpaired or self-supervised enhancement models. The "silence" attack is a disruption attack, and while the content-tampering extension is shown, the robustness of the specific phrase output against different TTS voices or languages is not deeply explored.
This work highlights a critical security vulnerability in passive audio processing pipelines. As speech enhancement becomes ubiquitous in real-time communication and voice assistants, the ability to silently disable or manipulate these services using natural audio inputs poses a significant risk. The findings urge the community to consider backdoor robustness in the design of speech enhancement models, particularly those deployed in safety-critical or privacy-sensitive applications. The paper introduces a novel backdoor attack for speech enhancement that uses clean audio as a natural trigger, demonstrating high effectiveness and stealthiness across multiple architectures and real-world scenarios. By shifting the trigger from artificial injections to natural high-SNR speech, the work addresses a realistic threat model for passive audio processing services, providing significant insights into the security vulnerabilities of current speech enhancement pipelines.
Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ρ=-0.068$, PESQ $ρ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.
Primary: GN Group
All Institutions: GN Group
The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
The paper introduces Output Divergence Rate (ODR), a metric designed to quantify the semantic impact of speech enhancement (SE) on downstream Large Language Model (LLM) intent classification. The methodology is sound, utilizing a cascaded pipeline of ASR (Whisper large-v3 and wav2vec2-large) followed by LLM classification (Gemini 2.5 Flash Lite). The approach correctly identifies a critical gap in current evaluation practices: standard audio quality metrics (PESQ, STOI) and transcription metrics (WER) do not correlate well with semantic task performance. The use of a closed-set intent classification task (SLURP) provides a controlled environment to measure divergence, although it limits the scope to specific command structures. The decomposition of divergence into "corrections" vs. "regressions" using ground truth labels is a rigorous addition that validates the metric's sensitivity to actual harm rather than random noise.
The experiments are comprehensive, covering five distinct enhancement conditions (Noisy, MetricGAN+, Echo, Echo+AEC, Dereverb) across 2,974 clips. The results are striking: MetricGAN+ doubles the ODR compared to unenhanced noisy speech despite improving PESQ, and unmitigated echo causes catastrophic failure (ODR 0.836) due to speaker substitution, a failure mode WER cannot capture. The replication across two architecturally distinct ASR models (attention-based vs. CTC) strengthens the findings, showing that the issue is not model-specific. The analysis of within-condition vs. pooled correlations is particularly valuable, demonstrating that while metrics separate conditions, they fail to predict per-clip divergence, which is crucial for deployment monitoring.
High. The authors release the full pipeline on GitHub. The use of standard, publicly available datasets (SLURP, DNS Challenge) and models (Whisper, MetricGAN+, Gemini API) ensures that the results are reproducible by other researchers. The prompt used for LLM classification is provided in the appendix, and hyperparameters for ASR and SE models are specified.
The study relies on simulated degradations (DNS Challenge) rather than real-world field recordings, which may differ in acoustic complexity. The LLM evaluation is limited to a single model family (Gemini) for the primary results, with only partial replication on a larger model (Gemini Pro). The task is restricted to English and closed-set intent classification, so findings may not generalize to open-ended generation or other languages. ODR requires a clean reference, limiting its use as an online monitor.
This paper has significant implications for the deployment of spoken AI systems. It challenges the common assumption that improving audio quality metrics (like PESQ) leads to better user experience in LLM-based assistants. It provides a new diagnostic tool (ODR) for developers to evaluate SE front-ends specifically for their impact on semantic understanding. The finding that standard metrics are poor predictors of LLM failure urges the field to adopt task-specific evaluation protocols for voice interfaces. The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.
Primary: University of Houston
All Institutions: University of Houston, Stevens Institute of Technology, Waseda University
[One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
The paper proposes PhysWave, a physics-guided latent diffusion model for text-to-First-Order Ambisonics (FOA) generation. The core methodological contribution is the integration of differentiable acoustic priors (spherical-harmonic direction consistency and inverse-square distance consistency) into the diffusion training objective. By decoding the predicted clean latent back to the waveform during training and applying these physical constraints, the model explicitly enforces spatial accuracy rather than relying solely on implicit data-driven learning. Additionally, the paper introduces a unified waypoint-caption representation that allows both natural language descriptions and precise parametric trajectories to condition the same model, bridging the gap between usability and precision.
The authors construct a 300K-clip synthetic FOA dataset, which is a significant resource contribution. Experiments include ablations on the physics losses, comparisons with state-of-the-art methods (ImmerseDiffusion, SonicMotion), and subjective listening tests. The results show that the physics priors significantly reduce angular error and improve distance consistency without degrading audio quality metrics (FAD, CLAP). The subjective evaluation with 14 participants confirms the perceptual benefits.
The paper provides detailed implementation details, including the VAE architecture, diffusion model hyperparameters, and the specific prompts used for the LLM parser. The release of the 300K dataset and the project website enhances reproducibility. However, the reliance on a specific LLM (Qwen3.5-4B) for parsing might introduce variability, though the deterministic post-processing helps.
The model currently focuses on single-source trajectories in a free-field setting, ignoring room effects like reverberation and occlusion. It also does not handle multi-source scenes. The synthetic nature of the dataset, while large, may not fully capture the complexity of real-world spatial audio recordings.
This work has significant potential for the gaming, film, and VR industries by enabling non-experts to create high-quality spatial audio via text. The physics-guided approach could be extended to other domains where physical constraints are known, such as video generation or scientific simulation. [One sentence main contribution]. The paper presents PhysWave, a physics-guided latent diffusion model that improves spatial consistency in text-to-FOA generation by incorporating differentiable acoustic priors for direction and distance, along with a unified control interface for natural language and parametric trajectories.
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erber’s Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.
Primary: National University of Singapore
All Institutions: National University of Singapore, Tencent, LIGHTSPEED, Nanyang Technological University, Independent Researcher, The Chinese University of Hong Kong, Shenzhen, Shenzhen Loop Area Institute
The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
The paper proposes "LoopTTS," a closed-loop TTS framework comprising a Filter, Judge (AudioLLM), and Refiner. The core methodological contribution is the Refiner, a fine-tuned TTS model (based on EmoVoice) that performs guided expressive re-synthesis conditioned on the initial defective audio, target text, and structured instructions from the AudioLLM Judge. The training methodology introduces a "Position-Weighted Cross-Entropy Loss" and a "Structural Alignment Loss" using Representational Similarity Matrices (RSMs) to focus supervision on sparse prosodic spans (stress/pause). The data construction pipeline ("Refiner-DB") uses contrastive AudioLLM annotation to generate weak supervision for word-level prosody. The approach is technically sound, addressing the specific gap of local prosodic defects that utterance-level metrics miss. The use of RSMs for span-level consistency is a creative application of representation learning to TTS fine-tuning.
The experiments are rigorous and well-structured. The authors evaluate the full pipeline on a subset of flagged utterances (200 samples) and a larger scaled evaluation (3,000 samples). They include strong baselines: open-loop re-generation (CosyVoice2, EmoVoice), global-only control, and a human-instruction upper bound. The use of a different AudioLLM (Qwen3-Omni) for the Judge stage demonstrates robustness. Subjective evaluations (MOS, MOS-I) are conducted by professional evaluators with reported ICC reliability. The results show consistent improvements in recovery quality and instruction-following fidelity (stress/pause) over baselines. The ablation studies effectively isolate the contributions of the position-weighting and structural alignment losses.
The paper provides high reproducibility. Code, data construction scripts, prompts, and model checkpoints are promised for release. The GitHub link is provided. Detailed hyperparameters, loss formulations, and dataset composition are included in the main text and appendices. The use of public datasets (ESD, RAVDESS, etc.) and standard models (CosyVoice2, Whisper) further enhances reproducibility.
The primary limitation is the reliance on a proprietary AudioLLM (Gemini-3-pro) for both data annotation and inference-time judging, which may limit accessibility and introduce potential biases. The evaluation is focused on English speech, and the framework's applicability to tonal languages is not tested. The "weak supervision" from AudioLLMs is acknowledged as imperfect, with moderate F1 scores for pause detection. The system is designed for offline batch processing, not streaming, due to the multi-stage latency.
This work has significant implications for the deployment of high-quality TTS systems in production environments where sporadic prosodic errors are unacceptable. It demonstrates a viable path for integrating LLM-based diagnostic capabilities into generative audio pipelines, moving beyond simple generation to generation-and-correction. The methodology of using AudioLLMs for scalable weak supervision of fine-grained prosodic features could be extended to other audio tasks, such as music production or sound effect design. The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
AudioSpan introduces a meticulously designed benchmark for long-form audio comprehension, addressing critical gaps in existing LALM evaluation. The core methodology revolves around two complementary question paths: Native QA and Anchor QA, both built upon a three-level cognitive taxonomy (Perception, Understanding, Reasoning). Native QA derives questions from the audio's inherent content, offering both multiple-choice (MCQ) and open-ended (OEQ) formats, with OEQs graded by an LLM ensemble using detailed rubrics that include semantic and temporal scores. Anchor QA is particularly innovative, injecting acoustic anchors into the audio to create verifiable ground truth and constructing a perception-to-reasoning chain of MCQs, scored only up to the first error to penalize shortcutting. This dual-path approach effectively combines broad coverage of natural content with precisely verifiable ground truth. The fully automated construction pipeline is a significant strength. Phase 1 generates a structured caption from the audio, involving ASR, semantic segmentation, acoustic captioning, dependency analysis, and caption assembly into evidence records. Phase 2 then generates QA pairs, with a sophisticated multi-level quality assurance system. This system includes evidence verification against the audio, a three-level solver (blind, text-only, full access) to filter out guessable or text-only answerable items, and a novel adversarial critic feedback loop that refines question generation by learning from past failure modes. This adaptive refinement mechanism is a key technical contribution, ensuring high-quality, audio-dependent questions at scale. The detailed rubric for OEQs, incorporating temporal grounding as a critical component, is also well-conceived and crucial for evaluating true comprehension.
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Amazon AGI Foundations
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
The paper introduces SpeechGym, a novel audio-native agentic environment designed for training voice agents via reinforcement learning. The core methodological contribution is the creation of an end-to-end trainable loop for multi-turn dialogue and tool use in speech, which is claimed to be a first. The environment integrates two omni-modal models (Qwen3-Omni-30B-A3B) – a frozen user model and a trainable agent model – allowing for native audio interaction without external ASR/TTS or proprietary APIs. This local setup is crucial for enabling gradient flow and affordable on-policy RL. The problem is formulated as a POMDP, where the agent decides between structured tool calls (textual) and variable-length speech responses. Keeping tool calls textual is a sensible design choice, isolating perceptual errors from reasoning errors. The environment inherits tasks, tools, databases, and success checks from the `tau2` text benchmark, ensuring modality is the primary variable. A new "Banking" domain is introduced, specifically stressing high-stakes numeric slots, which is a valuable addition for evaluating robustness to perceptual errors. The reward mechanism is based on `tau2`'s outcome check, a product of binary components, ensuring strict task completion. To address the severe sparsity of this reward in audio settings, the paper proposes a per-turn process shaping, crediting successful tool executions and penalizing failures. This is a classical reward shaping technique adapted effectively to the agentic setting. For efficient rollout collection, the authors leverage `vLLM-Omni`, serving both models and the LoRA adapter, which significantly speeds up training epochs and eliminates API costs, making online speech RL practical. The overall methodology is well-conceived, addressing key challenges in audio agent training.
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
Recent open text-audio contrastive models (CLAPs) are typically trained with LLM-generated captions derived from tag datasets or web search results, which tend to be accurate but expressively narrow. As a complementary source, we explore human-written album reviews, specifically expert reviews from AllMusic: they exist at scale and carry narrative cues, evaluative adjectives, and scene framing that other sources lack. Since raw reviews are too noisy for direct use as captions, we first build a caption corpus with 24,5346 samples via an LLM preprocessing pipeline that identifies descriptive musical quotes and rewrites them into training-ready captions. We find that album review supervision yields the largest retrieval gains on a human-written caption benchmark (Song Describer), particularly for complex queries that other existing caption datasets leave uncovered. In addition, we revisit the training recipe and show that SigReg regularization, which encourages an isotropic Gaussian distribution in the embedding space, improves MLP probing across classification tasks, as well as text-to-music retrieval. The resulting model outperforms open CLAP-style baselines on text-to-music retrieval, zero-shot classification, and most MLP probing tasks. We release the review-derived caption dataset and model weights to support future research.
Primary: Universitat Pompeu Fabra (UPF) / Music Technology Group
All Institutions: Universitat Pompeu Fabra (UPF), Music Technology Group, Barcelona Supercomputing Center
The paper presents a valuable contribution to music-text alignment by introducing a dataset derived from expert album reviews and demonstrating its efficacy in improving retrieval of complex musical descriptions. The integration of SigReg regularization and detailed ablation studies on training objectives adds technical depth, making it a solid contribution to the field of multimodal audio research.
The paper proposes a novel data-centric approach to training text-audio contrastive models (CLAPs) by leveraging expert-written album reviews from AllMusic. The core methodological contribution is an LLM-based preprocessing pipeline that extracts descriptive musical quotes from reviews and rewrites them into training-ready captions. This addresses a known gap in existing datasets (like MusicCaps) which rely on tag-to-text generation and lack narrative or evaluative depth. Additionally, the authors conduct an ablation study on training objectives, specifically investigating the integration of SigReg regularization (from LeJEPA) with the standard InfoNCE loss, and layer selection strategies for the audio encoder. The approach is sound and addresses a specific limitation in current multimodal music representation learning.
The evaluation is comprehensive, covering text-to-music retrieval (MusicCaps, Song Describer), zero-shot classification (GTZAN, FMA-Small), audio similarity (DimSim), and MLP probing (MTT, MTG-Jamendo). The results demonstrate that the proposed model outperforms open baselines (Laion-CLAP, TTMR++, CLaMP3) on most metrics, particularly on the Song Describer benchmark which contains complex, human-written captions. The analysis of query complexity (lexical-slice analysis) provides strong evidence that the review-derived data specifically benefits the retrieval of complex, narrative-driven queries. The inclusion of SigReg regularization shows consistent improvements in probing tasks, validating the theoretical benefit of isotropic embeddings for discriminative tasks.
The authors release the dataset, model weights, and code. The paper provides detailed descriptions of the LLM pipelines, hyperparameters, and training setups. The use of standard benchmarks ensures that results can be compared with future work. The reliance on LLMs for data generation introduces some stochasticity, but the pipeline is described in sufficient detail for replication.
The paper acknowledges that review-derived captions alone are insufficient and must be combined with other corpora. The reliance on LLMs for caption generation introduces potential biases or hallucinations, although the authors argue the risk is low based on manual inspection. The evaluation is primarily on English-language data, limiting generalizability. The "AllMusic" dataset is proprietary in its raw form, though the derived captions are released. The venue date "ISMIR 2026" is likely a typo for 2024 or 2025, but does not affect the technical assessment.
This work contributes to the open-source ecosystem for music information retrieval by providing a new dataset and model that capture more nuanced musical descriptions. It encourages the use of diverse textual sources for multimodal training. The release of weights and code supports further research in audio-language alignment. The paper presents a valuable contribution to music-text alignment by introducing a dataset derived from expert album reviews and demonstrating its efficacy in improving retrieval of complex musical descriptions. The integration of SigReg regularization and detailed ablation studies on training objectives adds technical depth, making it a solid contribution to the field of multimodal audio research.
Spatial audio vocoders are able to convert mel-spectrograms produced by generative models into spatial audio waveforms. Most neural vocoders are designed for monaural audio, and direct extensions to spatial audio can degrade spatial quality by ignoring inter-channel cues. We present CSAVocoder, a causal GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering. Our framework introduces a Spatial Adaptor that fuses multi-channel mel-spectrograms with dynamic source-listener pose information, together with a spatial consistency discriminator that supervises inter-channel cues. To meet real-time requirements, we design a strictly causal, stateful generator that supports efficient streaming inference with constant memory overhead. Experiments on large-scale spatial audio datasets show that CSAVocoder improves spatial fidelity at competitive audio quality and real-time performance.
Primary: Zhejiang University
All Institutions: Zhejiang University
[CSAVocoder presents a causal, GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering through a novel Spatial Adaptor and consistency discriminator, achieving real-time streaming performance with improved spatial fidelity.] This paper makes a solid technical contribution to the niche of spatial audio generation by addressing the critical gap of real-time, high-fidelity spatial vocoding. The integration of pose conditioning and inter-channel attention mechanisms provides a robust framework for preserving spatial cues in generated audio, advancing the state-of-the-art in immersive audio synthesis.
The paper proposes CSAVocoder, a causal GAN-based spatial audio vocoder. The core methodological contributions include a "Spatial Adaptor" that fuses multi-channel mel-spectrograms with dynamic source-listener pose information (using Fourier features and FiLM conditioning) and a "Spatial Consistency Discriminator" (SCD) that uses axial attention to model inter-channel relationships. The generator is redesigned to be strictly causal with stateful caching for streaming inference. The approach is technically sound, extending HiFi-GAN with specific modules for spatial conditioning and causality. However, the novelty is moderate; the combination of GAN-based vocoding with spatial conditioning is not entirely new (e.g., BinauralFlow, ISDrama), and the specific architectural tweaks (ShuffleUpsample, FiLM) are incremental adaptations of existing techniques to the spatial domain. The use of a dedicated SCD is a notable contribution to the adversarial training process for spatial audio.
The experiments are comprehensive, covering binaural and FOA formats. The authors compare against strong baselines including HiFi-GAN, Vocos, DiffWave, and ISDrama. They introduce new spatial consistency metrics (ANG COS, DIS COS) based on Spatial-AST embeddings, which is a valuable addition to the evaluation protocol. The results show improvements in spatial fidelity metrics while maintaining competitive audio quality (PESQ, MRSTFT) and achieving real-time performance (RTF < 1). The ablation studies effectively isolate the contributions of the Mel Adaptor, Position Adaptor, and SCD. The subjective evaluation (MOS-P, MOS-Q) supports the quantitative findings. The comparison with DSP and two-stage baselines clarifies the model's positioning.
The paper provides detailed architectural configurations, hyperparameters, and loss formulations in the appendix. The dataset sources (MRSSpeech, EasyCom, Spatial LibriSpeech, SoundSpaces) are clearly identified. The preprocessing steps are described. While the code is not linked, the level of detail provided is sufficient for reproduction by researchers in the field.
The authors acknowledge limitations in comparing against causal baselines due to implementation differences. They also note that the model is limited to binaural and FOA formats and does not address higher-order ambisonics or personalized HRTFs. The reliance on simulated data for some training sets may introduce domain gaps. The PESQ score is lower than non-causal baselines, which is expected but a trade-off for real-time capability.
This work contributes to the field of immersive audio generation, which has significant applications in VR/AR, gaming, and telepresence. By enabling real-time spatial audio synthesis, it lowers the barrier for interactive applications. The ethical considerations section is thorough, addressing privacy risks associated with pose data and potential misuse for deepfakes. [CSAVocoder presents a causal, GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering through a novel Spatial Adaptor and consistency discriminator, achieving real-time streaming performance with improved spatial fidelity.] This paper makes a solid technical contribution to the niche of spatial audio generation by addressing the critical gap of real-time, high-fidelity spatial vocoding. The integration of pose conditioning and inter-channel attention mechanisms provides a robust framework for preserving spatial cues in generated audio, advancing the state-of-the-art in immersive audio synthesis.
Fully few-shot class-incremental audio classification (FFCAC) requires recognizing new sound classes from only a handful of labeled examples per session, without forgetting previously learned classes and without any large base dataset. Existing methods typically freeze a pre-trained audio--language encoder and classify with point prototypes, but they suffer from significant performance degradation throughout the sessions due to generic feature representations. We propose SPECTRA, a framework built on a frozen encoder which adds three components. (i) a lightweight trainable adapter that calibrates the generic embeddings to the task; (ii) subspace feature replay, an exemplar-free anti-forgetting scheme that replays old classes by sampling from the low-rank subspace of their stored features; and (iii) a transductive optimal-transport refinement of prototypes at test time. Our central finding is that the subspace structure of the replay diminishes forgetting and outperforms naive Gaussian replay of equal variance. On three FFCAC benchmarks (NSynth-100, FSC-89, LS-100), SPECTRA improves average accuracy and reduces forgetting over current state-of-the-art methods, and our ablations statistically validate each component.
Primary: University of Haifa
All Institutions: University of Haifa, University of Stuttgart, IMPRS-IS
SPECTRA introduces a subspace-based exemplar-free replay mechanism and transductive optimal transport refinement to improve fully few-shot class-incremental audio classification, demonstrating consistent gains over the TAPE baseline on standard benchmarks. The technical contribution is solid and incremental, offering a practical solution to catastrophic forgetting in resource-constrained audio settings, though the novelty is moderate as it combines known techniques in a new configuration.
The paper proposes SPECTRA, a framework for Fully Few-Shot Class-Incremental Audio Classification (FFCAC). It builds upon the TAPE baseline by introducing three key components: (1) a lightweight trainable residual-FFN adapter to calibrate frozen audio-language model (ALM) embeddings, (2) Subspace Feature Replay, an exemplar-free anti-forgetting mechanism that synthesizes old-class features by sampling from the low-rank subspace of stored support features, and (3) Transductive Optimal Transport (OT) for prototype refinement at test time. The methodology is technically sound, leveraging established concepts from continual learning (replay, adapters) and few-shot learning (prototypical networks, optimal transport) but applying them in a specific, constrained audio setting. The core novelty lies in the subspace replay mechanism, which claims to preserve feature geometry better than isotropic Gaussian replay. The integration of these components into a frozen-encoder pipeline is a coherent engineering contribution.
The authors evaluate SPECTRA on three benchmarks: NSynth-100, FSC-89, and LS-100. They report improvements in Average Accuracy (AA) and Performance Drop (PD) over the TAPE baseline. The results are presented in tables showing per-session accuracy. The ablation studies are comprehensive, isolating the effects of the adapter, the subspace structure vs. Gaussian replay, and the OT refinement. The statistical validation (50 seeds) adds credibility. However, the improvements, while consistent, are moderate (e.g., +4.8% AA on NSynth-100). The comparison is primarily against TAPE, which is the current SOTA for this specific setting, but broader comparisons with other FFCAC methods or general continual learning techniques adapted for audio are limited. The "harder protocol" experiment adds some robustness to the claims.
The paper provides detailed descriptions of the components, including equations for the adapter, subspace sampling, and OT refinement. The experimental setup (datasets, protocols, hyperparameters like rank k=3, expansion r=3) is clearly stated. The use of a frozen PENGI encoder and TAPE's transform ensures a strong baseline for comparison. The code is not explicitly linked in the provided text, but the methodological details are sufficient for reproduction by experts in the field. The reliance on specific ALMs (PENGI) and the closed-form transform of TAPE limits direct comparability with methods using different backbones without re-implementation.
The paper acknowledges that OT increases forgetting slightly (PD), which is compensated by subspace replay. The method is dependent on the quality of the frozen ALM embeddings; if the ALM is poor, the adapter may struggle to calibrate effectively. The subspace rank k is a hyperparameter that requires tuning, although the paper shows it is not highly sensitive in the tested range. The method assumes that the low-rank subspace is a sufficient statistic for the class distribution, which may not hold for highly complex or multimodal audio classes with very few shots. The evaluation is limited to three datasets, and generalization to other audio domains (e.g., speech, environmental sounds beyond the specific benchmarks) is not extensively explored.
This work contributes to the field of efficient and continual audio AI, which is crucial for real-world applications like assisted living, wildlife monitoring, and interactive systems where data is scarce and models must adapt over time without forgetting. By enabling exemplar-free learning, it addresses privacy and storage concerns. The techniques could be extended to other modalities or more complex continual learning scenarios. SPECTRA introduces a subspace-based exemplar-free replay mechanism and transductive optimal transport refinement to improve fully few-shot class-incremental audio classification, demonstrating consistent gains over the TAPE baseline on standard benchmarks. The technical contribution is solid and incremental, offering a practical solution to catastrophic forgetting in resource-constrained audio settings, though the novelty is moderate as it combines known techniques in a new configuration.
We propose $\textit{recursive encoder and decoder}$ (RED) for building a single deep neural network (DNN) model that can separate multi-speaker mixtures containing unknown numbers of speakers and variable numbers of microphones arranged in an unknown geometry, a task that has not been studied yet. The decoder of RED recursively detects whether there are active speakers left and separates one speaker at a time. It is designed to be trained in an end-to-end fashion to improve separation performance. The encoder of RED recursively encodes each microphone channel of the input mixture, sequentially incorporating spatial cues. Combining both, the DNN can be trained to separate mixtures not only with unknown numbers of speakers but also with variable numbers of microphones, achieving state-of-the-art performance on multiple public datasets.
Primary: unknown (Affiliations not explicitly stated in text, authors are Fulin Wu and Zhong-Qiu Wang)
All Institutions: unknown
[One sentence main contribution]. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
The paper proposes REDnet, a recursive encoder-decoder architecture for speech separation. The core novelty lies in handling two variable dimensions simultaneously: the number of microphones (via a recursive encoder that aggregates channel features sequentially) and the number of speakers (via a recursive decoder that peels off one speaker at a time using a gating mechanism and a speaker detector). The methodology is technically sound, building upon established blocks (TF-GridNet) and concepts (Permutation Invariant Training, recursive separation). The recursive encoder design effectively addresses the variable microphone input problem by avoiding fixed-size pooling or explicit pairwise attention, offering a linear complexity solution. The recursive decoder's use of a gating mechanism to update the residual representation is a logical extension of prior recursive separation works, adapted for end-to-end training with specific loss functions to encourage orthogonality between separated speakers and residuals. While the combination is novel, the individual components are incremental improvements on existing paradigms (TAC for variable mics, OR-PIT/SepTDA for unknown speakers).
The experimental evaluation is comprehensive and rigorous. The authors decompose the problem to validate the Recursive Encoder (RE) and Recursive Decoder (RD) separately before presenting the full REDnet. They use standard public datasets (WSJ0, WHAM!, CHiME-4) and compare against strong baselines (TAC, FlexIO, MUSE, SR-CorrNet). The results show state-of-the-art performance on multiple metrics (SI-SDR, SDR, PESQ, STOI) across various configurations (variable mics, unknown speakers, noisy/reverberant conditions). The ablation studies on the RDnet component are particularly helpful in isolating the contributions of the gating mechanism, speaker interaction module, and loss functions. The evaluation on real-recorded signals (DNSMOS) adds practical value. The comparison with FlexIO is fair, noting that FlexIO assumes known speaker counts, whereas REDnet does not, making REDnet's performance gains even more significant.
The paper provides detailed architectural descriptions, including hyperparameters for the TF-GridNet blocks, optimizer settings, and loss weights. The dataset construction and preprocessing steps are clearly defined. The use of standard datasets and metrics facilitates reproduction. However, the specific implementation details of the "modified TF-GridNet blocks" (e.g., exact SwiGLU placement, kernel sizes) are described but might require careful reading to implement exactly. The code is not linked, which is a minor hindrance to immediate reproducibility, but the description is sufficient for a competent researcher.
The primary limitation is that the model's performance depends on the accuracy of the speaker detector in the recursive decoder. If the detector fails to stop (false positive) or stops too early (false negative), performance degrades. The paper reports high counting accuracy, but this is on controlled datasets. In highly noisy or reverberant real-world scenarios, detector robustness might be a challenge. Additionally, the recursive nature of the decoder implies that errors can propagate if the gating mechanism does not perfectly isolate the speaker, although the residual update mechanism mitigates this. The model assumes that the number of microphones is known at inference time (to process the input), which is standard, but the "unknown geometry" aspect is handled by the channel-agnostic aggregation, which is a strength but relies on the assumption that spatial cues are sufficiently captured by the recursive encoding.
This work contributes to the development of more robust and flexible audio processing systems, which are crucial for real-world applications like hearing aids, smart speakers, and teleconferencing systems where the number of speakers and microphones can vary. By unifying these capabilities into a single model, it reduces the need for multiple specialized models and simplifies deployment. The recursive approach is generalizable to other source separation tasks (e.g., music, sound event detection) and potentially to other modalities where variable input/output dimensions are a challenge. [One sentence main contribution]. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
Non-verbal vocalizations (NVs), such as laughter, coughs, and sighs, are essential for expressive TTS, but the effectiveness of preference optimization for NV generation remains poorly understood. We systematically study preference optimization for NV-capable TTS, focusing on preference signals, preference-pair construction, and DPO-based optimization objectives. We formulate an NV-aware character error rate (NV-CER) by treating NV tags as distinct output symbols and computing a weighted pinyin-based CER over both verbal and non-verbal content, enabling controllable optimization of NV realization without modifying the underlying optimization algorithm. Experiments on Emilia-NV and the augmented NV-Bench covering 18 NV types reveal how different design choices affect NV realization and lexical fidelity, and establish an effective setup using standard DPO. Objective, LLM-based, and human evaluations provide converging evidence for our findings, offering practical insights into NV-aware post-training for expressive TTS.
Primary: Nanyang Technological University
All Institutions: Nanyang Technological University, Nanjing University, National University of Singapore, LIGHTSPEED
This paper presents a systematic evaluation of DPO for non-verbal vocalization synthesis in TTS, introducing an NV-aware CER metric and demonstrating that NV-specific preference signals improve NV accuracy and overall synthesis quality over standard baselines.
The paper proposes a systematic study of Direct Preference Optimization (DPO) for Non-Verbal Vocalization (NV) synthesis in Text-to-Speech (TTS) systems. The core methodological contribution is the formulation of an NV-aware Character Error Rate (NV-CER) that treats NV tags as distinct symbols and computes a weighted edit distance over pinyin and NV tokens. This metric is used to construct preference pairs (preferred vs. rejected) for DPO. The authors also introduce a weighting mechanism ($w_{NV}$) to control the trade-off between NV realization and lexical fidelity. While the application of DPO to TTS is not entirely new, its systematic application and evaluation specifically for the nuanced task of NV generation, along with the specific NV-CER metric, provides a structured approach to a previously under-explored area. The methodology is sound but largely incremental, relying on standard DPO formulations and existing ASR models (SenseVoice) for preference signals.
The experiments are conducted on the Emilia-NV dataset and the NV-Bench. The authors evaluate various preference signals (ASR CER, NV-CER, NV-CER+UTMOS) and loss formulations (DPO, DPO+SFT). Results show that NV-CER-based preference signals outperform conventional ASR-CER and baseline SFT models in both objective metrics (NV-CER, PCER, UTMOS, DNSMOS) and LLM-based/human evaluations. The ablation studies on preference pair construction (Synthetic vs. Ground Truth) and NV-weight tuning provide valuable insights. However, the improvements in perceptual quality (UTMOS/DNSMOS) are modest, and the human evaluation shows high tie rates, suggesting that while NV accuracy improves, naturalness gains are less pronounced. The use of LLM-based evaluation adds a modern layer of assessment, though it correlates with objective metrics.
The paper provides sufficient implementation details, including the base model (CosyVoice2-0.5B), training hyperparameters (learning rates, epochs, batch sizes), and the specific NV-ASR model used (SenseVoice-Small fine-tuned). The dataset (Emilia-NV) and benchmark (NV-Bench) are publicly available or can be constructed as described. The code is not explicitly linked, but the methodology is clear enough for reproduction by researchers familiar with LLM-based TTS systems.
The primary limitation is that the preference signal relies entirely on an ASR model, which may not perfectly correlate with human perception of NV naturalness or correctness, especially for ambiguous NVs. The paper acknowledges that the base model rarely omits NVs, making "no-NV" a weak negative example, which limits the diversity of preference pairs. Furthermore, the improvements in perceptual quality are marginal, and the method does not fundamentally change the underlying TTS architecture, relying instead on post-training. The reliance on pinyin for Mandarin also limits immediate generalizability to other languages without adaptation.
This work contributes to the field of expressive TTS by providing practical guidelines for incorporating NVs into preference optimization pipelines. It helps bridge the gap between lexical TTS and more human-like, expressive speech synthesis. The NV-CER metric offers a standardized way to evaluate NV generation, which could facilitate future research in this area. The findings are relevant for developers of conversational AI agents and virtual assistants requiring more natural interaction capabilities. This paper presents a systematic evaluation of DPO for non-verbal vocalization synthesis in TTS, introducing an NV-aware CER metric and demonstrating that NV-specific preference signals improve NV accuracy and overall synthesis quality over standard baselines.
Spatial audio is a key component of immersive $360^\circ$ media, yet high-quality spatial capture remains limited in real-world speech-dominant scenes. We study visually guided First-Order Ambisonics (FOA) speech spatialization in the wild: given aligned $360^\circ$ video and an omnidirectional audio track, we recover the missing directional FOA components. To support this task, we introduce YT-SPEECH, a speech-oriented $360^\circ$ video-FOA dataset curated from YouTube. We propose a two-stage Localizer-Renderer framework, where an audio-visual segmentation backbone provides frame-wise spatial heatmaps and a conditional complex-domain U-Net reconstructs directional FOA signals from the omnidirectional channel. A confidence-based gating strategy stabilizes conditioning under ambiguous acoustic conditions. Experiments show improved reconstruction fidelity, spatial accuracy, and perceptual speech quality relative to ablated variants and prior approaches.
Primary: University of Surrey
All Institutions: University of Surrey
This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
The paper proposes a two-stage framework, Localizer-Renderer, for visually guided First-Order Ambisonics (FOA) speech spatialization. The Localizer utilizes an Audio-Visual Segmentation (AVS) backbone to generate spatial heatmaps from 360-degree video, which are then used as conditional priors for a complex-domain U-Net Renderer. The approach introduces a confidence-based gating mechanism to modulate the conditioning strength based on the entropy and peakiness of the spatial prior. While the integration of AVS for spatial grounding in audio generation is a logical extension of current multimodal trends, the core methodology relies on established components (U-Net, AVS backbones, FOA encoding) rather than introducing fundamentally new architectural primitives. The "confidence gating" is a heuristic modification rather than a deep theoretical innovation. The use of complex-valued networks for phase preservation is a known best practice in this specific subfield, not a novel contribution.
The authors curate a new dataset, YT-SPEECH, which is a valuable resource for the community, addressing the scarcity of paired 360-video and FOA speech data. The experimental evaluation includes ablation studies on the Localizer and Renderer components, as well as comparisons with analytic baselines (AmbiEnc, Pyroomacoustics) and a prior end-to-end model (SAG). The results show improvements in spatial accuracy (DOA error) and perceptual quality (PESQ, MOS). However, the subjective evaluation is limited to a small-scale listening test with only 9 participants, which reduces the statistical power of the perceptual claims. The comparison with SAG is conducted on SAG's own datasets, which is good for compatibility, but the lack of comparison with other recent state-of-the-art video-to-audio models (e.g., ViSAGe, OmniAudio) limits the assessment of relative performance in the broader landscape.
The paper provides sufficient detail on the architecture, loss functions, and training strategy (pretraining on Sphere360, fine-tuning on YT-SPEECH). The dataset curation pipeline is described in detail. However, the code is not explicitly linked in the text (only a demo URL is provided), and the specific hyperparameters for the AVS backbone fine-tuning are not fully enumerated, which may hinder exact reproduction. The reliance on proprietary or hard-to-access components (e.g., specific Whisper versions, YOLOv8-nano configurations) adds minor friction.
The authors acknowledge the limited scale of the dataset (8.9 hours) and reduced stability under overlapping sources and acoustically complex scenes. The method's performance is heavily dependent on the quality of the visual localization; if the AVS model fails to segment the speaker correctly, the spatial prior will be misleading, potentially degrading performance. The small sample size of the subjective listening test is a significant limitation for validating perceptual claims.
This work contributes to the field of immersive media and spatial audio, which has applications in virtual reality, telepresence, and accessible media for visually impaired users. The release of the YT-SPEECH dataset is a positive contribution to the community, enabling further research in visually guided spatial audio. However, the use of YouTube data raises potential copyright and privacy concerns regarding the training data, although the authors note manual inspection for quality and consistency. This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
The text contains content addressed to automated reviewers. Its score has been penalised, and the ranking below should not be trusted.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
Text addressed to an automated reviewer was found in this paper. The impact score shown has already been penalised for this, but treat the ranking with suspicion.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
Primary: Xiaohongshu
All Institutions: Xiaohongshu
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
The paper proposes FireRedAudio, a 9B-parameter unified audio-language model that decouples continuous input representations for understanding versus generation. The core methodological contribution is the use of two distinct pathways: a dedicated Audio Encoder (initialized from Whisper-large-v3) for understanding tasks (ASR, audio QA) and a RedAE-based continuous autoencoder pathway for speech generation (TTS, editing). The generation pathway utilizes a Flow-Matching DiT conditioned on LLM hidden states to generate RedAE latents, which are then decoded to waveforms. The authors argue that understanding benefits from compact, semantic features while generation requires reconstructible, fine-grained features, and that fusing them degrades performance. The training strategy involves a five-stage progressive curriculum, starting with adapter alignment, then encoder adaptation, unified mid-training, multitask post-training with CoT, and finally long-context extension. This approach is technically sound and addresses a known tension in unified multimodal models. The use of flow-matching for continuous latent generation is a modern and effective choice compared to standard diffusion or autoregressive discrete token prediction.
The evaluation covers a broad suite of tasks: ASR (LibriSpeech, FLEURS, etc.), Audio Understanding (MMAU, MMSU), Zero-shot TTS (Seed-TTS-Eval), Instruct TTS (InstructTTSEval), and Speech Editing (Ming-Freeform-Audio-Edit). The results are competitive or leading on most benchmarks. Notably, the model achieves strong performance in Instruct TTS instruction following and significant improvements in speech editing over prior work (Ming-UniAudio-Edit). The long-form audio understanding capability (up to 1 hour) with second-level timestamp accuracy is a strong practical contribution. The ablation studies and comparisons are comprehensive. The use of an LLM-as-a-judge for temporal grounding evaluation is a reasonable proxy but introduces potential bias; however, the strict/content metrics provide some robustness.
The paper provides detailed architectural descriptions, training stages, loss functions, and hyperparameters (learning rates, warmup steps). The code is made available on GitHub. The use of open-source components (Qwen3.5-9B, Whisper-large-v3) aids reproducibility. The RedAE model is described in detail, including its hybrid GAN and distillation loss. The main limitation for perfect reproducibility is the specific data mixtures and the proprietary nature of some training data (e.g., specific Instruct TTS datasets), though the sampling strategy is well-documented.
The model relies on a large 9B parameter backbone, which implies significant computational requirements for training and inference compared to smaller specialized models. The decoupled representation, while effective, adds architectural complexity and potentially increases inference latency due to the need to manage two distinct pathways and the DiT generation loop. The performance on very long-form audio (1 hour) might still suffer from context window limitations or memory constraints during inference, although the paper claims success. The reliance on a frozen teacher for RedAE distillation means the final RedAE latents are bounded by the teacher's capabilities.
FireRedAudio contributes to the field of unified multimodal AI, demonstrating that decoupled representations can effectively balance the conflicting objectives of understanding and generation. This has implications for developing more capable and efficient voice assistants, content creation tools, and accessibility technologies. The ability to perform high-quality speech editing and TTS with natural language instructions opens up new creative and practical applications. However, like all generative audio models, it raises concerns about deepfakes and misuse for impersonation, which the authors do not explicitly address in the provided text. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.