Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.
Primary: Future Academy, JD
All Institutions: Future Academy, JD
Analysis unavailable
Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Amazon AGI Foundations
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
The paper introduces SpeechGym, a novel audio-native agentic environment designed for training voice agents via reinforcement learning. The core methodological contribution is the creation of an end-to-end trainable loop for multi-turn dialogue and tool use in speech, which is claimed to be a first. The environment integrates two omni-modal models (Qwen3-Omni-30B-A3B) – a frozen user model and a trainable agent model – allowing for native audio interaction without external ASR/TTS or proprietary APIs. This local setup is crucial for enabling gradient flow and affordable on-policy RL. The problem is formulated as a POMDP, where the agent decides between structured tool calls (textual) and variable-length speech responses. Keeping tool calls textual is a sensible design choice, isolating perceptual errors from reasoning errors. The environment inherits tasks, tools, databases, and success checks from the `tau2` text benchmark, ensuring modality is the primary variable. A new "Banking" domain is introduced, specifically stressing high-stakes numeric slots, which is a valuable addition for evaluating robustness to perceptual errors. The reward mechanism is based on `tau2`'s outcome check, a product of binary components, ensuring strict task completion. To address the severe sparsity of this reward in audio settings, the paper proposes a per-turn process shaping, crediting successful tool executions and penalizing failures. This is a classical reward shaping technique adapted effectively to the agentic setting. For efficient rollout collection, the authors leverage `vLLM-Omni`, serving both models and the LoRA adapter, which significantly speeds up training epochs and eliminates API costs, making online speech RL practical. The overall methodology is well-conceived, addressing key challenges in audio agent training.
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
AudioSpan introduces a meticulously designed benchmark for long-form audio comprehension, addressing critical gaps in existing LALM evaluation. The core methodology revolves around two complementary question paths: Native QA and Anchor QA, both built upon a three-level cognitive taxonomy (Perception, Understanding, Reasoning). Native QA derives questions from the audio's inherent content, offering both multiple-choice (MCQ) and open-ended (OEQ) formats, with OEQs graded by an LLM ensemble using detailed rubrics that include semantic and temporal scores. Anchor QA is particularly innovative, injecting acoustic anchors into the audio to create verifiable ground truth and constructing a perception-to-reasoning chain of MCQs, scored only up to the first error to penalize shortcutting. This dual-path approach effectively combines broad coverage of natural content with precisely verifiable ground truth. The fully automated construction pipeline is a significant strength. Phase 1 generates a structured caption from the audio, involving ASR, semantic segmentation, acoustic captioning, dependency analysis, and caption assembly into evidence records. Phase 2 then generates QA pairs, with a sophisticated multi-level quality assurance system. This system includes evidence verification against the audio, a three-level solver (blind, text-only, full access) to filter out guessable or text-only answerable items, and a novel adversarial critic feedback loop that refines question generation by learning from past failure modes. This adaptive refinement mechanism is a key technical contribution, ensuring high-quality, audio-dependent questions at scale. The detailed rubric for OEQs, incorporating temporal grounding as a critical component, is also well-conceived and crucial for evaluating true comprehension.
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Yonsei University, University of Seoul
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
The paper introduces a hierarchical benchmark (HEAR) grounded in Erber’s Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.
Primary: National University of Singapore
All Institutions: National University of Singapore, Tencent, LIGHTSPEED, Nanyang Technological University, Independent Researcher, The Chinese University of Hong Kong, Shenzhen, Shenzhen Loop Area Institute
The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
The paper proposes "LoopTTS," a closed-loop TTS framework comprising a Filter, Judge (AudioLLM), and Refiner. The core methodological contribution is the Refiner, a fine-tuned TTS model (based on EmoVoice) that performs guided expressive re-synthesis conditioned on the initial defective audio, target text, and structured instructions from the AudioLLM Judge. The training methodology introduces a "Position-Weighted Cross-Entropy Loss" and a "Structural Alignment Loss" using Representational Similarity Matrices (RSMs) to focus supervision on sparse prosodic spans (stress/pause). The data construction pipeline ("Refiner-DB") uses contrastive AudioLLM annotation to generate weak supervision for word-level prosody. The approach is technically sound, addressing the specific gap of local prosodic defects that utterance-level metrics miss. The use of RSMs for span-level consistency is a creative application of representation learning to TTS fine-tuning.
The experiments are rigorous and well-structured. The authors evaluate the full pipeline on a subset of flagged utterances (200 samples) and a larger scaled evaluation (3,000 samples). They include strong baselines: open-loop re-generation (CosyVoice2, EmoVoice), global-only control, and a human-instruction upper bound. The use of a different AudioLLM (Qwen3-Omni) for the Judge stage demonstrates robustness. Subjective evaluations (MOS, MOS-I) are conducted by professional evaluators with reported ICC reliability. The results show consistent improvements in recovery quality and instruction-following fidelity (stress/pause) over baselines. The ablation studies effectively isolate the contributions of the position-weighting and structural alignment losses.
The paper provides high reproducibility. Code, data construction scripts, prompts, and model checkpoints are promised for release. The GitHub link is provided. Detailed hyperparameters, loss formulations, and dataset composition are included in the main text and appendices. The use of public datasets (ESD, RAVDESS, etc.) and standard models (CosyVoice2, Whisper) further enhances reproducibility.
The primary limitation is the reliance on a proprietary AudioLLM (Gemini-3-pro) for both data annotation and inference-time judging, which may limit accessibility and introduce potential biases. The evaluation is focused on English speech, and the framework's applicability to tonal languages is not tested. The "weak supervision" from AudioLLMs is acknowledged as imperfect, with moderate F1 scores for pause detection. The system is designed for offline batch processing, not streaming, due to the multi-stage latency.
This work has significant implications for the deployment of high-quality TTS systems in production environments where sporadic prosodic errors are unacceptable. It demonstrates a viable path for integrating LLM-based diagnostic capabilities into generative audio pipelines, moving beyond simple generation to generation-and-correction. The methodology of using AudioLLMs for scalable weak supervision of fine-grained prosodic features could be extended to other audio tasks, such as music production or sound effect design. The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
AudioSpan introduces a meticulously designed benchmark for long-form audio comprehension, addressing critical gaps in existing LALM evaluation. The core methodology revolves around two complementary question paths: Native QA and Anchor QA, both built upon a three-level cognitive taxonomy (Perception, Understanding, Reasoning). Native QA derives questions from the audio's inherent content, offering both multiple-choice (MCQ) and open-ended (OEQ) formats, with OEQs graded by an LLM ensemble using detailed rubrics that include semantic and temporal scores. Anchor QA is particularly innovative, injecting acoustic anchors into the audio to create verifiable ground truth and constructing a perception-to-reasoning chain of MCQs, scored only up to the first error to penalize shortcutting. This dual-path approach effectively combines broad coverage of natural content with precisely verifiable ground truth. The fully automated construction pipeline is a significant strength. Phase 1 generates a structured caption from the audio, involving ASR, semantic segmentation, acoustic captioning, dependency analysis, and caption assembly into evidence records. Phase 2 then generates QA pairs, with a sophisticated multi-level quality assurance system. This system includes evidence verification against the audio, a three-level solver (blind, text-only, full access) to filter out guessable or text-only answerable items, and a novel adversarial critic feedback loop that refines question generation by learning from past failure modes. This adaptive refinement mechanism is a key technical contribution, ensuring high-quality, audio-dependent questions at scale. The detailed rubric for OEQs, incorporating temporal grounding as a critical component, is also well-conceived and crucial for evaluating true comprehension.
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, Amazon AGI Foundations
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
The paper introduces SpeechGym, a novel audio-native agentic environment designed for training voice agents via reinforcement learning. The core methodological contribution is the creation of an end-to-end trainable loop for multi-turn dialogue and tool use in speech, which is claimed to be a first. The environment integrates two omni-modal models (Qwen3-Omni-30B-A3B) – a frozen user model and a trainable agent model – allowing for native audio interaction without external ASR/TTS or proprietary APIs. This local setup is crucial for enabling gradient flow and affordable on-policy RL. The problem is formulated as a POMDP, where the agent decides between structured tool calls (textual) and variable-length speech responses. Keeping tool calls textual is a sensible design choice, isolating perceptual errors from reasoning errors. The environment inherits tasks, tools, databases, and success checks from the `tau2` text benchmark, ensuring modality is the primary variable. A new "Banking" domain is introduced, specifically stressing high-stakes numeric slots, which is a valuable addition for evaluating robustness to perceptual errors. The reward mechanism is based on `tau2`'s outcome check, a product of binary components, ensuring strict task completion. To address the severe sparsity of this reward in audio settings, the paper proposes a per-turn process shaping, crediting successful tool executions and penalizing failures. This is a classical reward shaping technique adapted effectively to the agentic setting. For efficient rollout collection, the authors leverage `vLLM-Omni`, serving both models and the LoRA adapter, which significantly speeds up training epochs and eliminates API costs, making online speech RL practical. The overall methodology is well-conceived, addressing key challenges in audio agent training.
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
Recent open text-audio contrastive models (CLAPs) are typically trained with LLM-generated captions derived from tag datasets or web search results, which tend to be accurate but expressively narrow. As a complementary source, we explore human-written album reviews, specifically expert reviews from AllMusic: they exist at scale and carry narrative cues, evaluative adjectives, and scene framing that other sources lack. Since raw reviews are too noisy for direct use as captions, we first build a caption corpus with 24,5346 samples via an LLM preprocessing pipeline that identifies descriptive musical quotes and rewrites them into training-ready captions. We find that album review supervision yields the largest retrieval gains on a human-written caption benchmark (Song Describer), particularly for complex queries that other existing caption datasets leave uncovered. In addition, we revisit the training recipe and show that SigReg regularization, which encourages an isotropic Gaussian distribution in the embedding space, improves MLP probing across classification tasks, as well as text-to-music retrieval. The resulting model outperforms open CLAP-style baselines on text-to-music retrieval, zero-shot classification, and most MLP probing tasks. We release the review-derived caption dataset and model weights to support future research.
Primary: Universitat Pompeu Fabra (UPF) / Music Technology Group
All Institutions: Universitat Pompeu Fabra (UPF), Music Technology Group, Barcelona Supercomputing Center
The paper presents a valuable contribution to music-text alignment by introducing a dataset derived from expert album reviews and demonstrating its efficacy in improving retrieval of complex musical descriptions. The integration of SigReg regularization and detailed ablation studies on training objectives adds technical depth, making it a solid contribution to the field of multimodal audio research.
The paper proposes a novel data-centric approach to training text-audio contrastive models (CLAPs) by leveraging expert-written album reviews from AllMusic. The core methodological contribution is an LLM-based preprocessing pipeline that extracts descriptive musical quotes from reviews and rewrites them into training-ready captions. This addresses a known gap in existing datasets (like MusicCaps) which rely on tag-to-text generation and lack narrative or evaluative depth. Additionally, the authors conduct an ablation study on training objectives, specifically investigating the integration of SigReg regularization (from LeJEPA) with the standard InfoNCE loss, and layer selection strategies for the audio encoder. The approach is sound and addresses a specific limitation in current multimodal music representation learning.
The evaluation is comprehensive, covering text-to-music retrieval (MusicCaps, Song Describer), zero-shot classification (GTZAN, FMA-Small), audio similarity (DimSim), and MLP probing (MTT, MTG-Jamendo). The results demonstrate that the proposed model outperforms open baselines (Laion-CLAP, TTMR++, CLaMP3) on most metrics, particularly on the Song Describer benchmark which contains complex, human-written captions. The analysis of query complexity (lexical-slice analysis) provides strong evidence that the review-derived data specifically benefits the retrieval of complex, narrative-driven queries. The inclusion of SigReg regularization shows consistent improvements in probing tasks, validating the theoretical benefit of isotropic embeddings for discriminative tasks.
The authors release the dataset, model weights, and code. The paper provides detailed descriptions of the LLM pipelines, hyperparameters, and training setups. The use of standard benchmarks ensures that results can be compared with future work. The reliance on LLMs for data generation introduces some stochasticity, but the pipeline is described in sufficient detail for replication.
The paper acknowledges that review-derived captions alone are insufficient and must be combined with other corpora. The reliance on LLMs for caption generation introduces potential biases or hallucinations, although the authors argue the risk is low based on manual inspection. The evaluation is primarily on English-language data, limiting generalizability. The "AllMusic" dataset is proprietary in its raw form, though the derived captions are released. The venue date "ISMIR 2026" is likely a typo for 2024 or 2025, but does not affect the technical assessment.
This work contributes to the open-source ecosystem for music information retrieval by providing a new dataset and model that capture more nuanced musical descriptions. It encourages the use of diverse textual sources for multimodal training. The release of weights and code supports further research in audio-language alignment. The paper presents a valuable contribution to music-text alignment by introducing a dataset derived from expert album reviews and demonstrating its efficacy in improving retrieval of complex musical descriptions. The integration of SigReg regularization and detailed ablation studies on training objectives adds technical depth, making it a solid contribution to the field of multimodal audio research.
Spatial audio vocoders are able to convert mel-spectrograms produced by generative models into spatial audio waveforms. Most neural vocoders are designed for monaural audio, and direct extensions to spatial audio can degrade spatial quality by ignoring inter-channel cues. We present CSAVocoder, a causal GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering. Our framework introduces a Spatial Adaptor that fuses multi-channel mel-spectrograms with dynamic source-listener pose information, together with a spatial consistency discriminator that supervises inter-channel cues. To meet real-time requirements, we design a strictly causal, stateful generator that supports efficient streaming inference with constant memory overhead. Experiments on large-scale spatial audio datasets show that CSAVocoder improves spatial fidelity at competitive audio quality and real-time performance.
Primary: Zhejiang University
All Institutions: Zhejiang University
[CSAVocoder presents a causal, GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering through a novel Spatial Adaptor and consistency discriminator, achieving real-time streaming performance with improved spatial fidelity.] This paper makes a solid technical contribution to the niche of spatial audio generation by addressing the critical gap of real-time, high-fidelity spatial vocoding. The integration of pose conditioning and inter-channel attention mechanisms provides a robust framework for preserving spatial cues in generated audio, advancing the state-of-the-art in immersive audio synthesis.
The paper proposes CSAVocoder, a causal GAN-based spatial audio vocoder. The core methodological contributions include a "Spatial Adaptor" that fuses multi-channel mel-spectrograms with dynamic source-listener pose information (using Fourier features and FiLM conditioning) and a "Spatial Consistency Discriminator" (SCD) that uses axial attention to model inter-channel relationships. The generator is redesigned to be strictly causal with stateful caching for streaming inference. The approach is technically sound, extending HiFi-GAN with specific modules for spatial conditioning and causality. However, the novelty is moderate; the combination of GAN-based vocoding with spatial conditioning is not entirely new (e.g., BinauralFlow, ISDrama), and the specific architectural tweaks (ShuffleUpsample, FiLM) are incremental adaptations of existing techniques to the spatial domain. The use of a dedicated SCD is a notable contribution to the adversarial training process for spatial audio.
The experiments are comprehensive, covering binaural and FOA formats. The authors compare against strong baselines including HiFi-GAN, Vocos, DiffWave, and ISDrama. They introduce new spatial consistency metrics (ANG COS, DIS COS) based on Spatial-AST embeddings, which is a valuable addition to the evaluation protocol. The results show improvements in spatial fidelity metrics while maintaining competitive audio quality (PESQ, MRSTFT) and achieving real-time performance (RTF < 1). The ablation studies effectively isolate the contributions of the Mel Adaptor, Position Adaptor, and SCD. The subjective evaluation (MOS-P, MOS-Q) supports the quantitative findings. The comparison with DSP and two-stage baselines clarifies the model's positioning.
The paper provides detailed architectural configurations, hyperparameters, and loss formulations in the appendix. The dataset sources (MRSSpeech, EasyCom, Spatial LibriSpeech, SoundSpaces) are clearly identified. The preprocessing steps are described. While the code is not linked, the level of detail provided is sufficient for reproduction by researchers in the field.
The authors acknowledge limitations in comparing against causal baselines due to implementation differences. They also note that the model is limited to binaural and FOA formats and does not address higher-order ambisonics or personalized HRTFs. The reliance on simulated data for some training sets may introduce domain gaps. The PESQ score is lower than non-causal baselines, which is expected but a trade-off for real-time capability.
This work contributes to the field of immersive audio generation, which has significant applications in VR/AR, gaming, and telepresence. By enabling real-time spatial audio synthesis, it lowers the barrier for interactive applications. The ethical considerations section is thorough, addressing privacy risks associated with pose data and potential misuse for deepfakes. [CSAVocoder presents a causal, GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering through a novel Spatial Adaptor and consistency discriminator, achieving real-time streaming performance with improved spatial fidelity.] This paper makes a solid technical contribution to the niche of spatial audio generation by addressing the critical gap of real-time, high-fidelity spatial vocoding. The integration of pose conditioning and inter-channel attention mechanisms provides a robust framework for preserving spatial cues in generated audio, advancing the state-of-the-art in immersive audio synthesis.
Fully few-shot class-incremental audio classification (FFCAC) requires recognizing new sound classes from only a handful of labeled examples per session, without forgetting previously learned classes and without any large base dataset. Existing methods typically freeze a pre-trained audio--language encoder and classify with point prototypes, but they suffer from significant performance degradation throughout the sessions due to generic feature representations. We propose SPECTRA, a framework built on a frozen encoder which adds three components. (i) a lightweight trainable adapter that calibrates the generic embeddings to the task; (ii) subspace feature replay, an exemplar-free anti-forgetting scheme that replays old classes by sampling from the low-rank subspace of their stored features; and (iii) a transductive optimal-transport refinement of prototypes at test time. Our central finding is that the subspace structure of the replay diminishes forgetting and outperforms naive Gaussian replay of equal variance. On three FFCAC benchmarks (NSynth-100, FSC-89, LS-100), SPECTRA improves average accuracy and reduces forgetting over current state-of-the-art methods, and our ablations statistically validate each component.
Primary: University of Haifa
All Institutions: University of Haifa, University of Stuttgart, IMPRS-IS
SPECTRA introduces a subspace-based exemplar-free replay mechanism and transductive optimal transport refinement to improve fully few-shot class-incremental audio classification, demonstrating consistent gains over the TAPE baseline on standard benchmarks. The technical contribution is solid and incremental, offering a practical solution to catastrophic forgetting in resource-constrained audio settings, though the novelty is moderate as it combines known techniques in a new configuration.
The paper proposes SPECTRA, a framework for Fully Few-Shot Class-Incremental Audio Classification (FFCAC). It builds upon the TAPE baseline by introducing three key components: (1) a lightweight trainable residual-FFN adapter to calibrate frozen audio-language model (ALM) embeddings, (2) Subspace Feature Replay, an exemplar-free anti-forgetting mechanism that synthesizes old-class features by sampling from the low-rank subspace of stored support features, and (3) Transductive Optimal Transport (OT) for prototype refinement at test time. The methodology is technically sound, leveraging established concepts from continual learning (replay, adapters) and few-shot learning (prototypical networks, optimal transport) but applying them in a specific, constrained audio setting. The core novelty lies in the subspace replay mechanism, which claims to preserve feature geometry better than isotropic Gaussian replay. The integration of these components into a frozen-encoder pipeline is a coherent engineering contribution.
The authors evaluate SPECTRA on three benchmarks: NSynth-100, FSC-89, and LS-100. They report improvements in Average Accuracy (AA) and Performance Drop (PD) over the TAPE baseline. The results are presented in tables showing per-session accuracy. The ablation studies are comprehensive, isolating the effects of the adapter, the subspace structure vs. Gaussian replay, and the OT refinement. The statistical validation (50 seeds) adds credibility. However, the improvements, while consistent, are moderate (e.g., +4.8% AA on NSynth-100). The comparison is primarily against TAPE, which is the current SOTA for this specific setting, but broader comparisons with other FFCAC methods or general continual learning techniques adapted for audio are limited. The "harder protocol" experiment adds some robustness to the claims.
The paper provides detailed descriptions of the components, including equations for the adapter, subspace sampling, and OT refinement. The experimental setup (datasets, protocols, hyperparameters like rank k=3, expansion r=3) is clearly stated. The use of a frozen PENGI encoder and TAPE's transform ensures a strong baseline for comparison. The code is not explicitly linked in the provided text, but the methodological details are sufficient for reproduction by experts in the field. The reliance on specific ALMs (PENGI) and the closed-form transform of TAPE limits direct comparability with methods using different backbones without re-implementation.
The paper acknowledges that OT increases forgetting slightly (PD), which is compensated by subspace replay. The method is dependent on the quality of the frozen ALM embeddings; if the ALM is poor, the adapter may struggle to calibrate effectively. The subspace rank k is a hyperparameter that requires tuning, although the paper shows it is not highly sensitive in the tested range. The method assumes that the low-rank subspace is a sufficient statistic for the class distribution, which may not hold for highly complex or multimodal audio classes with very few shots. The evaluation is limited to three datasets, and generalization to other audio domains (e.g., speech, environmental sounds beyond the specific benchmarks) is not extensively explored.
This work contributes to the field of efficient and continual audio AI, which is crucial for real-world applications like assisted living, wildlife monitoring, and interactive systems where data is scarce and models must adapt over time without forgetting. By enabling exemplar-free learning, it addresses privacy and storage concerns. The techniques could be extended to other modalities or more complex continual learning scenarios. SPECTRA introduces a subspace-based exemplar-free replay mechanism and transductive optimal transport refinement to improve fully few-shot class-incremental audio classification, demonstrating consistent gains over the TAPE baseline on standard benchmarks. The technical contribution is solid and incremental, offering a practical solution to catastrophic forgetting in resource-constrained audio settings, though the novelty is moderate as it combines known techniques in a new configuration.
We propose $\textit{recursive encoder and decoder}$ (RED) for building a single deep neural network (DNN) model that can separate multi-speaker mixtures containing unknown numbers of speakers and variable numbers of microphones arranged in an unknown geometry, a task that has not been studied yet. The decoder of RED recursively detects whether there are active speakers left and separates one speaker at a time. It is designed to be trained in an end-to-end fashion to improve separation performance. The encoder of RED recursively encodes each microphone channel of the input mixture, sequentially incorporating spatial cues. Combining both, the DNN can be trained to separate mixtures not only with unknown numbers of speakers but also with variable numbers of microphones, achieving state-of-the-art performance on multiple public datasets.
Primary: unknown (Affiliations not explicitly stated in text, authors are Fulin Wu and Zhong-Qiu Wang)
All Institutions: unknown
[One sentence main contribution]. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
The paper proposes REDnet, a recursive encoder-decoder architecture for speech separation. The core novelty lies in handling two variable dimensions simultaneously: the number of microphones (via a recursive encoder that aggregates channel features sequentially) and the number of speakers (via a recursive decoder that peels off one speaker at a time using a gating mechanism and a speaker detector). The methodology is technically sound, building upon established blocks (TF-GridNet) and concepts (Permutation Invariant Training, recursive separation). The recursive encoder design effectively addresses the variable microphone input problem by avoiding fixed-size pooling or explicit pairwise attention, offering a linear complexity solution. The recursive decoder's use of a gating mechanism to update the residual representation is a logical extension of prior recursive separation works, adapted for end-to-end training with specific loss functions to encourage orthogonality between separated speakers and residuals. While the combination is novel, the individual components are incremental improvements on existing paradigms (TAC for variable mics, OR-PIT/SepTDA for unknown speakers).
The experimental evaluation is comprehensive and rigorous. The authors decompose the problem to validate the Recursive Encoder (RE) and Recursive Decoder (RD) separately before presenting the full REDnet. They use standard public datasets (WSJ0, WHAM!, CHiME-4) and compare against strong baselines (TAC, FlexIO, MUSE, SR-CorrNet). The results show state-of-the-art performance on multiple metrics (SI-SDR, SDR, PESQ, STOI) across various configurations (variable mics, unknown speakers, noisy/reverberant conditions). The ablation studies on the RDnet component are particularly helpful in isolating the contributions of the gating mechanism, speaker interaction module, and loss functions. The evaluation on real-recorded signals (DNSMOS) adds practical value. The comparison with FlexIO is fair, noting that FlexIO assumes known speaker counts, whereas REDnet does not, making REDnet's performance gains even more significant.
The paper provides detailed architectural descriptions, including hyperparameters for the TF-GridNet blocks, optimizer settings, and loss weights. The dataset construction and preprocessing steps are clearly defined. The use of standard datasets and metrics facilitates reproduction. However, the specific implementation details of the "modified TF-GridNet blocks" (e.g., exact SwiGLU placement, kernel sizes) are described but might require careful reading to implement exactly. The code is not linked, which is a minor hindrance to immediate reproducibility, but the description is sufficient for a competent researcher.
The primary limitation is that the model's performance depends on the accuracy of the speaker detector in the recursive decoder. If the detector fails to stop (false positive) or stops too early (false negative), performance degrades. The paper reports high counting accuracy, but this is on controlled datasets. In highly noisy or reverberant real-world scenarios, detector robustness might be a challenge. Additionally, the recursive nature of the decoder implies that errors can propagate if the gating mechanism does not perfectly isolate the speaker, although the residual update mechanism mitigates this. The model assumes that the number of microphones is known at inference time (to process the input), which is standard, but the "unknown geometry" aspect is handled by the channel-agnostic aggregation, which is a strength but relies on the assumption that spatial cues are sufficiently captured by the recursive encoding.
This work contributes to the development of more robust and flexible audio processing systems, which are crucial for real-world applications like hearing aids, smart speakers, and teleconferencing systems where the number of speakers and microphones can vary. By unifying these capabilities into a single model, it reduces the need for multiple specialized models and simplifies deployment. The recursive approach is generalizable to other source separation tasks (e.g., music, sound event detection) and potentially to other modalities where variable input/output dimensions are a challenge. [One sentence main contribution]. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
Non-verbal vocalizations (NVs), such as laughter, coughs, and sighs, are essential for expressive TTS, but the effectiveness of preference optimization for NV generation remains poorly understood. We systematically study preference optimization for NV-capable TTS, focusing on preference signals, preference-pair construction, and DPO-based optimization objectives. We formulate an NV-aware character error rate (NV-CER) by treating NV tags as distinct output symbols and computing a weighted pinyin-based CER over both verbal and non-verbal content, enabling controllable optimization of NV realization without modifying the underlying optimization algorithm. Experiments on Emilia-NV and the augmented NV-Bench covering 18 NV types reveal how different design choices affect NV realization and lexical fidelity, and establish an effective setup using standard DPO. Objective, LLM-based, and human evaluations provide converging evidence for our findings, offering practical insights into NV-aware post-training for expressive TTS.
Primary: Nanyang Technological University
All Institutions: Nanyang Technological University, Nanjing University, National University of Singapore, LIGHTSPEED
This paper presents a systematic evaluation of DPO for non-verbal vocalization synthesis in TTS, introducing an NV-aware CER metric and demonstrating that NV-specific preference signals improve NV accuracy and overall synthesis quality over standard baselines.
The paper proposes a systematic study of Direct Preference Optimization (DPO) for Non-Verbal Vocalization (NV) synthesis in Text-to-Speech (TTS) systems. The core methodological contribution is the formulation of an NV-aware Character Error Rate (NV-CER) that treats NV tags as distinct symbols and computes a weighted edit distance over pinyin and NV tokens. This metric is used to construct preference pairs (preferred vs. rejected) for DPO. The authors also introduce a weighting mechanism ($w_{NV}$) to control the trade-off between NV realization and lexical fidelity. While the application of DPO to TTS is not entirely new, its systematic application and evaluation specifically for the nuanced task of NV generation, along with the specific NV-CER metric, provides a structured approach to a previously under-explored area. The methodology is sound but largely incremental, relying on standard DPO formulations and existing ASR models (SenseVoice) for preference signals.
The experiments are conducted on the Emilia-NV dataset and the NV-Bench. The authors evaluate various preference signals (ASR CER, NV-CER, NV-CER+UTMOS) and loss formulations (DPO, DPO+SFT). Results show that NV-CER-based preference signals outperform conventional ASR-CER and baseline SFT models in both objective metrics (NV-CER, PCER, UTMOS, DNSMOS) and LLM-based/human evaluations. The ablation studies on preference pair construction (Synthetic vs. Ground Truth) and NV-weight tuning provide valuable insights. However, the improvements in perceptual quality (UTMOS/DNSMOS) are modest, and the human evaluation shows high tie rates, suggesting that while NV accuracy improves, naturalness gains are less pronounced. The use of LLM-based evaluation adds a modern layer of assessment, though it correlates with objective metrics.
The paper provides sufficient implementation details, including the base model (CosyVoice2-0.5B), training hyperparameters (learning rates, epochs, batch sizes), and the specific NV-ASR model used (SenseVoice-Small fine-tuned). The dataset (Emilia-NV) and benchmark (NV-Bench) are publicly available or can be constructed as described. The code is not explicitly linked, but the methodology is clear enough for reproduction by researchers familiar with LLM-based TTS systems.
The primary limitation is that the preference signal relies entirely on an ASR model, which may not perfectly correlate with human perception of NV naturalness or correctness, especially for ambiguous NVs. The paper acknowledges that the base model rarely omits NVs, making "no-NV" a weak negative example, which limits the diversity of preference pairs. Furthermore, the improvements in perceptual quality are marginal, and the method does not fundamentally change the underlying TTS architecture, relying instead on post-training. The reliance on pinyin for Mandarin also limits immediate generalizability to other languages without adaptation.
This work contributes to the field of expressive TTS by providing practical guidelines for incorporating NVs into preference optimization pipelines. It helps bridge the gap between lexical TTS and more human-like, expressive speech synthesis. The NV-CER metric offers a standardized way to evaluate NV generation, which could facilitate future research in this area. The findings are relevant for developers of conversational AI agents and virtual assistants requiring more natural interaction capabilities. This paper presents a systematic evaluation of DPO for non-verbal vocalization synthesis in TTS, introducing an NV-aware CER metric and demonstrating that NV-specific preference signals improve NV accuracy and overall synthesis quality over standard baselines.
Spatial audio is a key component of immersive $360^\circ$ media, yet high-quality spatial capture remains limited in real-world speech-dominant scenes. We study visually guided First-Order Ambisonics (FOA) speech spatialization in the wild: given aligned $360^\circ$ video and an omnidirectional audio track, we recover the missing directional FOA components. To support this task, we introduce YT-SPEECH, a speech-oriented $360^\circ$ video-FOA dataset curated from YouTube. We propose a two-stage Localizer-Renderer framework, where an audio-visual segmentation backbone provides frame-wise spatial heatmaps and a conditional complex-domain U-Net reconstructs directional FOA signals from the omnidirectional channel. A confidence-based gating strategy stabilizes conditioning under ambiguous acoustic conditions. Experiments show improved reconstruction fidelity, spatial accuracy, and perceptual speech quality relative to ablated variants and prior approaches.
Primary: University of Surrey
All Institutions: University of Surrey
This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
The paper proposes a two-stage framework, Localizer-Renderer, for visually guided First-Order Ambisonics (FOA) speech spatialization. The Localizer utilizes an Audio-Visual Segmentation (AVS) backbone to generate spatial heatmaps from 360-degree video, which are then used as conditional priors for a complex-domain U-Net Renderer. The approach introduces a confidence-based gating mechanism to modulate the conditioning strength based on the entropy and peakiness of the spatial prior. While the integration of AVS for spatial grounding in audio generation is a logical extension of current multimodal trends, the core methodology relies on established components (U-Net, AVS backbones, FOA encoding) rather than introducing fundamentally new architectural primitives. The "confidence gating" is a heuristic modification rather than a deep theoretical innovation. The use of complex-valued networks for phase preservation is a known best practice in this specific subfield, not a novel contribution.
The authors curate a new dataset, YT-SPEECH, which is a valuable resource for the community, addressing the scarcity of paired 360-video and FOA speech data. The experimental evaluation includes ablation studies on the Localizer and Renderer components, as well as comparisons with analytic baselines (AmbiEnc, Pyroomacoustics) and a prior end-to-end model (SAG). The results show improvements in spatial accuracy (DOA error) and perceptual quality (PESQ, MOS). However, the subjective evaluation is limited to a small-scale listening test with only 9 participants, which reduces the statistical power of the perceptual claims. The comparison with SAG is conducted on SAG's own datasets, which is good for compatibility, but the lack of comparison with other recent state-of-the-art video-to-audio models (e.g., ViSAGe, OmniAudio) limits the assessment of relative performance in the broader landscape.
The paper provides sufficient detail on the architecture, loss functions, and training strategy (pretraining on Sphere360, fine-tuning on YT-SPEECH). The dataset curation pipeline is described in detail. However, the code is not explicitly linked in the text (only a demo URL is provided), and the specific hyperparameters for the AVS backbone fine-tuning are not fully enumerated, which may hinder exact reproduction. The reliance on proprietary or hard-to-access components (e.g., specific Whisper versions, YOLOv8-nano configurations) adds minor friction.
The authors acknowledge the limited scale of the dataset (8.9 hours) and reduced stability under overlapping sources and acoustically complex scenes. The method's performance is heavily dependent on the quality of the visual localization; if the AVS model fails to segment the speaker correctly, the spatial prior will be misleading, potentially degrading performance. The small sample size of the subjective listening test is a significant limitation for validating perceptual claims.
This work contributes to the field of immersive media and spatial audio, which has applications in virtual reality, telepresence, and accessible media for visually impaired users. The release of the YT-SPEECH dataset is a positive contribution to the community, enabling further research in visually guided spatial audio. However, the use of YouTube data raises potential copyright and privacy concerns regarding the training data, although the authors note manual inspection for quality and consistency. This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
The text contains content addressed to automated reviewers. Its score has been penalised, and the ranking below should not be trusted.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
Text addressed to an automated reviewer was found in this paper. The impact score shown has already been penalised for this, but treat the ranking with suspicion.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
Primary: Xiaohongshu
All Institutions: Xiaohongshu
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
The paper proposes FireRedAudio, a 9B-parameter unified audio-language model that decouples continuous input representations for understanding versus generation. The core methodological contribution is the use of two distinct pathways: a dedicated Audio Encoder (initialized from Whisper-large-v3) for understanding tasks (ASR, audio QA) and a RedAE-based continuous autoencoder pathway for speech generation (TTS, editing). The generation pathway utilizes a Flow-Matching DiT conditioned on LLM hidden states to generate RedAE latents, which are then decoded to waveforms. The authors argue that understanding benefits from compact, semantic features while generation requires reconstructible, fine-grained features, and that fusing them degrades performance. The training strategy involves a five-stage progressive curriculum, starting with adapter alignment, then encoder adaptation, unified mid-training, multitask post-training with CoT, and finally long-context extension. This approach is technically sound and addresses a known tension in unified multimodal models. The use of flow-matching for continuous latent generation is a modern and effective choice compared to standard diffusion or autoregressive discrete token prediction.
The evaluation covers a broad suite of tasks: ASR (LibriSpeech, FLEURS, etc.), Audio Understanding (MMAU, MMSU), Zero-shot TTS (Seed-TTS-Eval), Instruct TTS (InstructTTSEval), and Speech Editing (Ming-Freeform-Audio-Edit). The results are competitive or leading on most benchmarks. Notably, the model achieves strong performance in Instruct TTS instruction following and significant improvements in speech editing over prior work (Ming-UniAudio-Edit). The long-form audio understanding capability (up to 1 hour) with second-level timestamp accuracy is a strong practical contribution. The ablation studies and comparisons are comprehensive. The use of an LLM-as-a-judge for temporal grounding evaluation is a reasonable proxy but introduces potential bias; however, the strict/content metrics provide some robustness.
The paper provides detailed architectural descriptions, training stages, loss functions, and hyperparameters (learning rates, warmup steps). The code is made available on GitHub. The use of open-source components (Qwen3.5-9B, Whisper-large-v3) aids reproducibility. The RedAE model is described in detail, including its hybrid GAN and distillation loss. The main limitation for perfect reproducibility is the specific data mixtures and the proprietary nature of some training data (e.g., specific Instruct TTS datasets), though the sampling strategy is well-documented.
The model relies on a large 9B parameter backbone, which implies significant computational requirements for training and inference compared to smaller specialized models. The decoupled representation, while effective, adds architectural complexity and potentially increases inference latency due to the need to manage two distinct pathways and the DiT generation loop. The performance on very long-form audio (1 hour) might still suffer from context window limitations or memory constraints during inference, although the paper claims success. The reliance on a frozen teacher for RedAE distillation means the final RedAE latents are bounded by the teacher's capabilities.
FireRedAudio contributes to the field of unified multimodal AI, demonstrating that decoupled representations can effectively balance the conflicting objectives of understanding and generation. This has implications for developing more capable and efficient voice assistants, content creation tools, and accessibility technologies. The ability to perform high-quality speech editing and TTS with natural language instructions opens up new creative and practical applications. However, like all generative audio models, it raises concerns about deepfakes and misuse for impersonation, which the authors do not explicitly address in the provided text. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.
Primary: Future Academy, JD
All Institutions: Future Academy, JD
Analysis unavailable
We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video, environmental sound, music and speech. We organize interaction around camera intent: in first-person scenes, it specifies observer motion, while in third-person scenes, camera--character dynamics are learned from data without view-specific controllers. Discrete commands and continuous poses are mapped to a shared metric-scale relative 6-DoF trajectory, with dataset-level calibration preserving motion magnitude across heterogeneous data. To jointly learn audio-visual generation and trajectory control, we construct a complementary data engine and adopt progressive training followed by autoregressive post-training for long-horizon generation. Extensive evaluations show that \model achieves strong trajectory following and high visual quality on public world-model benchmarks, supporting both first- and third-person interaction across varied subjects, and maintaining synchronized environmental sound and speech over long-horizon generation.
Primary: JD (Joy Future Academy)
All Institutions: Joy Future Academy, JD, Beihang University, Stanford University
EchoWM presents a comprehensive framework for omnimodal world modeling by unifying camera-centric navigation with joint audio-visual generation through a novel data engine and progressive training curriculum. The technical contribution of mapping heterogeneous control signals to a shared metric-scale trajectory and integrating this via UCPE is a significant step towards more controllable and interactive generative media, although the reliance on recovered geometry and the limitation to 720p resolution remain notable constraints.
The paper proposes "EchoWM," an omnimodal world model that jointly generates 720p video, environmental sound, music, and speech conditioned on a continuous 6-DoF camera trajectory. The core methodological contribution lies in the "camera intent" interface, which unifies first-person and third-person navigation by mapping discrete commands and continuous poses to a shared relative trajectory. The authors introduce a "complementary data engine" combining internal gameplay, internet gameplay, UE simulation, and web video, processed through separate audio-visual and geometry paths. A key technical detail is the use of Unified Camera Positional Encoding (UCPE) to inject relative camera geometry into the video transformer's attention mechanism, avoiding the need for separate camera encoders. The training strategy is progressive: Audio-Visual Continued Pretraining (AV-CPT), Action-SFT (freezing the backbone to learn trajectory conditioning), Joint-FT, and Autoregressive Post-Training (using Self-Gradient Forcing) for long-horizon generation. The approach is technically sound, addressing the distributional mismatch between audio-visual quality and control precision by curating specific data mixtures for each training stage.
The paper claims state-of-the-art performance on public benchmarks (WBench Navigation, SANA-WM-Bench). It emphasizes the ability to maintain synchronized audio and video over long horizons. However, the provided text is truncated, limiting the assessment of the full experimental results, ablation studies, and quantitative metrics (e.g., FAD, CLAP score, UTMOS, or human evaluation scores). The reliance on "public world-model benchmarks" suggests a focus on visual and control consistency, but the specific metrics for audio quality and synchronization fidelity are not detailed in the excerpt. The claim of "strong trajectory following" is supported by the controlled data engine, but the evaluation of speech synchronization specifically is less clear from the text.
The paper provides detailed descriptions of the data processing pipeline, including metric pose recovery using ViPE, VGGT-Omega, and MoGe-2. It specifies the use of Qwen3-Omni and Gemini-3-Pro for annotation. The training stages and loss functions are mathematically defined. However, the exact hyperparameters, the specific versions of the base models (e.g., the pretrained joint audio-visual diffusion transformer), and the code for the data engine are not included in the text. The reliance on proprietary internal gameplay data and specific annotation pipelines may hinder exact reproducibility without significant effort.
The model is limited to 720p resolution, which is modest compared to recent high-fidelity video generators. The interaction is restricted to camera/navigation intent; semantic actor actions (attacking, jumping) are not modeled. The audio generation, while synchronized, is conditioned on visual trajectory rather than direct acoustic modeling, which may limit the fidelity of complex acoustic environments. The reliance on recovered camera poses from internet video introduces potential noise, although quality filtering is applied. The truncation of the paper prevents a full assessment of failure modes or edge cases in long-horizon generation.
EchoWM contributes to the field of generative AI by enabling more immersive and interactive multimedia experiences. The ability to generate synchronized audio-visual content in response to navigation could impact gaming, virtual reality, and simulation. However, the generation of realistic speech and environmental sounds raises concerns about deepfakes and misinformation, particularly if the technology is misused to create deceptive audio-visual content. The paper does not explicitly discuss safety mitigations. EchoWM presents a comprehensive framework for omnimodal world modeling by unifying camera-centric navigation with joint audio-visual generation through a novel data engine and progressive training curriculum. The technical contribution of mapping heterogeneous control signals to a shared metric-scale trajectory and integrating this via UCPE is a significant step towards more controllable and interactive generative media, although the reliance on recovered geometry and the limitation to 720p resolution remain notable constraints.
Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.
Primary: LIGHTSPEED
All Institutions: LIGHTSPEED, National University of Singapore, Nanyang Technological University, Independent Researcher
EmoTra-TTS makes a significant contribution to expressive TTS by introducing a novel framework for smooth intra-utterance emotion transitions, combining synthetic data generation with a robust dual-stage conditioning mechanism. The rigorous evaluation and clear ablation studies demonstrate the effectiveness of the proposed methods, although generalization beyond the training dataset remains an area for future work.
The paper addresses the underexplored problem of intra-utterance emotion transitions in TTS, moving beyond static emotion conditioning. The proposed EmoTra-TTS framework is built on three pillars: (1) a synthetic data generation pipeline using multi-pass flow blending to create frame-aligned transition samples, addressing data scarcity; (2) dual-stage conditioning where an LLM handles prosodic planning via temporal VAD tokens, and a flow decoder realizes acoustic features via frame-level VAD embeddings; and (3) a direction-magnitude decoupled injection mechanism using LayerNorm and a fixed scale to prevent content degradation when injecting emotion signals into a frozen decoder. The methodology is sound and addresses specific failure modes of previous approaches (e.g., content collapse, magnitude mismatch) with rigorous ablation studies. The separation of prosodic planning and acoustic realization is a logical extension of current LLM-based TTS architectures.
The experimental evaluation is comprehensive, including MOS-based metrics (Quality, Emotion, Transition), speaker similarity, WER, and pairwise preference tests against four open-source baselines and two commercial systems. The results show significant improvements in transition smoothness (MOS-Tra) and overall preference. The inclusion of an objective metric (Prosodic Jerk Ratio) strengthens the findings. The ablation studies effectively isolate the contributions of each component. However, the evaluation is limited to the EmoVoice-DB dataset, which restricts claims about generalization to other speakers or languages. The use of synthetic data for training, while effective, introduces a domain gap that is partially mitigated by the quality filtering but remains a limitation.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code and demo are publicly available. The synthetic data generation pipeline is clearly described, allowing for reproduction of the training data. The use of a frozen backbone (CosyVoice2) and specific injection mechanisms makes the implementation straightforward to replicate.
The primary limitation is the reliance on the EmoVoice-DB dataset, which has limited speaker diversity and language coverage. The paper acknowledges that zero-shot generalization to unseen speakers is not demonstrated. Additionally, the emotion control stability for subtle VAD differences is noted as inconsistent. The piecewise linear interpolation of VAD trajectories is a simplification of natural emotion dynamics.
EmoTra-TTS contributes to the field of affective computing and expressive speech synthesis. By enabling smooth, controllable emotion transitions, it opens up new possibilities for storytelling, dubbing, and interactive virtual agents. The ethical statement addresses potential misuse, such as generating deceptive audio, and emphasizes responsible use. The work encourages further research into dynamic emotion modeling in speech. EmoTra-TTS makes a significant contribution to expressive TTS by introducing a novel framework for smooth intra-utterance emotion transitions, combining synthetic data generation with a robust dual-stage conditioning mechanism. The rigorous evaluation and clear ablation studies demonstrate the effectiveness of the proposed methods, although generalization beyond the training dataset remains an area for future work.
Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues over multiple audio inputs, requiring models to identify degradation types, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and explain low-level acoustic phenomena in natural language. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. MRMAD reveals an important yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
Primary: Northeastern University
All Institutions: Northeastern University, Bose Corporation, Stony Brook University
[One sentence main contribution]. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
The paper introduces MRMAD, a benchmark designed to evaluate Large Audio-Language Models (LALMs) on their ability to perceive and reason about audio degradation. The methodology involves constructing a dataset of 8,400 multiple-choice questions across three domains (speech, music, sound) and nine degradation types (e.g., noise, reverberation, compression). The evaluation is framed as multi-turn dialogues where models must identify degradation types, compare severity between two clips, or rank three clips by severity. The approach is methodologically sound for its purpose: it isolates low-level perceptual capabilities from high-level semantic understanding by using clean references and controlled degradations. The multi-turn design is a clever solution to the problem of comparing multiple audio inputs without temporal localization confusion. However, the novelty is moderate as it is essentially a specialized benchmark addition to the growing list of audio evaluation suites, rather than proposing a new model architecture or fundamental algorithm.
The experimental setup is comprehensive, evaluating 18 representative LALMs, including open-source models (SALMONN, Qwen2-Audio, Audio Flamingo) and closed-source/reasoning models (GPT-4o, Gemini, Qwen3-Omni). The results clearly demonstrate that current state-of-the-art models perform near random chance on degradation type identification and struggle significantly with severity comparison and ranking. This provides a valuable diagnostic baseline. The evaluation covers a wide range of model sizes and types, strengthening the generalizability of the findings. The use of balanced sampling and specific distractor generation for multiple-choice questions adds rigor to the evaluation protocol.
The paper provides a clear data construction pipeline, specifying source datasets (VCTK, LibriSpeech, MUSDB18, etc.), degradation parameters, and question generation templates. The code and dataset are made available via a GitHub link, which significantly enhances reproducibility. The multi-turn format and template-based prompt generation are described in sufficient detail for others to replicate the evaluation.
A key limitation is the reliance on synthetic degradations applied to high-quality source audio. While this allows for controlled severity levels, it may not fully capture the complexity of real-world audio degradation, which often involves complex, non-linear interactions between noise sources and signal characteristics. Additionally, the multiple-choice format, while efficient for large-scale evaluation, may not fully capture the nuanced reasoning capabilities of models compared to open-ended generation tasks. The exclusion of certain degradations for specific domains (e.g., echo in music) is justified but limits the universality of the benchmark. Finally, the evaluation is limited to 18 models; as the field evolves rapidly, the benchmark will need continuous updating.
This work has significant implications for the development of robust audio-language models. By highlighting the gap in low-level audio perception, it directs future research towards improving model robustness to real-world acoustic conditions. This is crucial for applications in accessibility (e.g., audio description for visually impaired users in noisy environments), audio restoration, and high-fidelity communication systems. The benchmark serves as a necessary tool for the community to track progress in this underexplored area. [One sentence main contribution]. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].