We address the problem of encoding room impulse responses (RIRs) into high-order Ambisonics (HOA) representations from arbitrary and potentially insufficient or incomplete microphone array measurements. This task is fundamentally ill-posed for microphone arrays with limited spatial capture capabilities, such as irregular or sparse arrays, as classical linear methods fail to reconstruct high-order spatial detail. We introduce a diffusion-based generative framework that models the statistical properties of HOA RIRs. This enables device-agnostic encoding from arbitrary microphone arrays, potentially unseen during data measurement. Our approach incorporates a posterior sampling procedure that enforces consistency between the estimated signals and the measurements while plausibly reconstructing spatial information that is unobservable from the limited measurements alone. Experiments on simulated data demonstrate that our method outperforms linear and neural baselines, achieving accurate HOA RIR estimation up to 12th order. A listening test with binaural renderings, including both simulated and measured RIRs, further confirms that the proposed method yields higher perceptual similarity to reference Ambisonics RIRs than all baselines. The flexibility and accuracy of the proposed framework opens new possibilities for scalable acoustics simulations.
Primary: Reality Labs Research, Meta
All Institutions: Reality Labs Research, Meta, Aalto University, Acoustics Lab
This paper presents a significant advancement in spatial audio processing by effectively leveraging diffusion models to solve the ill-posed problem of high-order Ambisonics encoding from sparse measurements, achieving state-of-the-art perceptual and objective performance.
The paper proposes a novel application of diffusion models to the ill-posed inverse problem of encoding Room Impulse Responses (RIRs) into High-Order Ambisonics (HOA) from sparse or irregular microphone arrays. The core methodological contribution is the integration of a device-agnostic diffusion prior with a posterior sampling procedure that enforces data consistency via a range-projected likelihood term. The use of a hybrid time-frequency/time-domain architecture (NCSN++ backbone with a U-Net refinement stage for early reflections) is a well-reasoned design choice given the distinct temporal characteristics of RIRs. The introduction of a compressed spectrogram distance metric for the likelihood guidance is a technically sound innovation to emphasize weak high-order components. The formulation separates the device-agnostic prior from the device-specific likelihood, which is a strong theoretical foundation for generalization.
The experimental evaluation is rigorous and comprehensive. The authors utilize two datasets: a large-scale internal FDTD simulation dataset and the public Treble-10 dataset. They compare against three strong baselines: Linear Least-Squares, a Time-Dependent Neural Encoder, and a Conditional Diffusion model specific to the device. The inclusion of both objective metrics (EDC, NPM) and a subjective listening test (MUSHRA-style with binaural rendering) provides robust validation. The results demonstrate clear superiority in both perceptual similarity and objective metrics, particularly in preserving early reflections and late reverberation tails. The ablation studies effectively isolate the contribution of the range-projection constraint and the hybrid architecture.
The paper provides sufficient detail for reproduction, including dataset descriptions, model architectures (NCSN++ based), training hyperparameters (AdamW, learning rate, batch size), and specific implementation details like the compressed spectrogram definition. The use of standard libraries (PyTorch) and well-known architectures aids reproducibility. However, the reliance on an internal FDTD dataset for the primary training results limits independent verification of the scale of the results, though the public Treble-10 results offer some ground truth.
The primary limitation is the distribution gap between simulated training data and real-world measured RIRs, which the authors acknowledge and demonstrate leads to performance drops on measured data (Eigenmike-64). The method assumes a highly accurate Array Transfer Function (ATF), which may not hold in real-world deployments with calibration errors. Additionally, the iterative sampling process is computationally intensive, making it less suitable for real-time applications without significant acceleration techniques. The evaluation is currently limited to a single device configuration (Aria Glasses) for the device-agnostic claim, though the method is theoretically general.
This work has significant implications for spatial audio processing, particularly for Virtual Reality (VR) and Augmented Reality (AR) applications where scalable and accurate acoustic simulation is crucial. By enabling high-quality HOA encoding from sparse arrays, it lowers the barrier for capturing spatial audio with wearable devices. It also facilitates the generation of large-scale training data for spatial audio models. The device-agnostic nature of the approach promotes interoperability across different hardware platforms. This paper presents a significant advancement in spatial audio processing by effectively leveraging diffusion models to solve the ill-posed problem of high-order Ambisonics encoding from sparse measurements, achieving state-of-the-art perceptual and objective performance.
Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.
Primary: Universitat Pompeu Fabra (Music Technology Group)
All Institutions: Universitat Pompeu Fabra, Music Technology Group
This paper makes a significant contribution to the understanding of representation quality in music foundation models by introducing a systematic layer-wise analysis and a novel pitch-transposition equivariance metric, demonstrating that intrinsic properties can effectively guide layer selection, particularly for tonal tasks where standard metrics fail.
The paper presents a rigorous and systematic layer-wise analysis of 12 music foundation models across three distinct pre-training paradigms (masked, autoregressive, contrastive). The methodology is sound, employing a comprehensive suite of label-free intrinsic metrics (Intrinsic Dimension, Curvature, Anisotropy, Effective Rank, LiDAR, InfoNCE) to characterize representation geometry. The key methodological contribution is the introduction of Pitch-Transposition Equivariance (PTE), a novel metric designed to capture tonal structure that standard geometric metrics miss. The approach of correlating these intrinsic properties with downstream probe performance across 15 diverse MIR tasks provides a robust framework for understanding representation quality without relying solely on task-specific supervision.
The experimental evaluation is extensive, covering a wide range of downstream tasks including tonal, rhythmic, timbral, semantic, and similarity tasks. The results are well-supported, demonstrating that while standard metrics correlate well with non-tonal tasks, they fail for tonal tasks, necessitating the new PTE metric. The paper effectively demonstrates that intrinsic metrics can serve as effective proxies for layer selection, often outperforming trainable multi-layer fusion methods, particularly in low-data regimes. The analysis of depth-wise trends across different model families provides valuable insights into how representation properties evolve.
The paper provides significant detail on the models, datasets (MTG-Jamendo, GiantSteps, NSynth, etc.), and evaluation protocols. The code and extended results are available on the project page, enhancing reproducibility. The use of standard datasets and publicly available models facilitates independent verification.
The study is correlational; it identifies properties associated with good performance but does not establish causality. The metrics are evaluated on frozen representations, so their utility for guiding pre-training or fine-tuning is not directly addressed. The analysis is limited to 12 models, which, while diverse, may not cover all architectural variations. The PTE metric, while promising, is specific to tonal tasks and may not generalize to other musical attributes.
This work provides practical guidelines for selecting layers in music foundation models, potentially reducing the computational cost of model evaluation and deployment. It advances the theoretical understanding of representation quality in audio models, bridging the gap between geometric analysis and practical MIR performance. The findings are relevant to the broader community working with self-supervised audio representations. This paper makes a significant contribution to the understanding of representation quality in music foundation models by introducing a systematic layer-wise analysis and a novel pitch-transposition equivariance metric, demonstrating that intrinsic properties can effectively guide layer selection, particularly for tonal tasks where standard metrics fail.
Automatic video dubbing in the wild remains fundamentally limited by two competing constraints: hierarchical methods depend on brittle, multi-stage preprocessing pipelines that severely restrict data scalability and practical deployment, while holistic approaches operating on uncropped video suffer from weak temporal alignment and speaker-utterance ambiguity in multi-speaker settings. To overcome these limitations, we propose CineDub, a unified diffusion-based model that achieves precise multi-speaker dialogue dubbing directly from uncropped videos, without face cropping or speaker diarization. Central to our approach is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm, where holistic visual representations and a semantic-bundled transcription format are encoded independently, yet implicitly coupled through cross-modal training to resolve speaker ambiguity and enable precise multi-speaker multi-turn dialogue dubbing. Building on the unified temporal cues captured by holistic visual features, we further extend CineDub to joint speech and audio generation. We introduce an Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation, and a decoupled textual branch control mechanism to resolve cross-prompt interference during simultaneous generation. We also release two in-the-wild benchmarks, CineDub-Multi for multi-speaker dialogue dubbing and CineDub-SA for video-to-speech-and-audio (V2SA) generation, to enable evaluation under realistic conditions. Experiments show that CineDub achieves state-of-the-art results on established single-speaker dubbing and video-to-audio benchmarks while excelling in multi-speaker dialogue dubbing and acoustically coherent joint generation.
Primary: Monash University
All Institutions: Monash University, University of Chinese Academy of Sciences, Tsinghua University
CineDub presents a robust and innovative approach to multi-speaker video dubbing by effectively decoupling holistic visual conditioning from semantic transcription, achieving state-of-the-art results and providing valuable new benchmarks for the community.
The paper proposes CineDub, a unified diffusion-based framework for end-to-end video dubbing that operates on uncropped videos, addressing the limitations of hierarchical methods (which rely on brittle preprocessing like face cropping and diarization) and holistic methods (which suffer from speaker-utterance ambiguity). The core technical contribution is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm. This involves using SynchFormer features as a holistic visual condition, which the authors argue captures both event-level audio-visual correspondence and fine-grained lip-sync cues through emergent attention mechanisms. To resolve speaker ambiguity, they introduce a "semantic-bundled transcription" format, where speaker descriptions are coupled with transcript segments, encoded by a pre-trained LLM (Gemma-T5). The framework extends to joint speech and audio generation (V2SA), introducing two key mechanisms: Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation by training on audio first, then speech, and a decoupled textual branch control mechanism to prevent cross-prompt interference between speech and audio conditions in the diffusion transformer. The approach is technically sound, leveraging recent advances in diffusion transformers and multimodal alignment, but the novelty lies primarily in the specific conditioning strategies and curriculum learning design rather than fundamental architectural changes.
The authors evaluate CineDub on single-speaker dubbing (GRID, CHEM), multi-speaker dubbing (new CineDub-Multi benchmark), video-to-audio (VGGSound), and joint V2SA (new CineDub-SA benchmark). Results show CineDub outperforms hierarchical baselines (like HPMDubbing, Speak2Dub) and holistic baselines (DeepDubber, DeepAudio) on most metrics, particularly in zero-shot voice cloning and multi-speaker scenarios. The introduction of two new benchmarks, CineDub-Multi and CineDub-SA, is a significant contribution, addressing the lack of realistic, in-the-wild evaluation data for multi-speaker and joint generation tasks. The ablation studies effectively demonstrate the necessity of the semantic-bundled transcription, the ALC curriculum, and the decoupled branches. The use of standard metrics (WER, LSE-D, UTMOS, FDVGG) is appropriate, though the reliance on embedding-based metrics for audio quality (FDVGG, KL) has known limitations regarding perceptual fidelity, which the authors partially address with UTMOS.
The paper provides detailed descriptions of the model architecture, training stages, and data processing pipelines. It mentions the use of pre-trained models (SynchFormer, Gemma-T5, CLIP) and specific datasets (VGGSound, AudioSet, SpeakerVid-5M). The release of the CineDub-Multi and CineDub-SA benchmarks enhances reproducibility for future work. However, the code is not explicitly linked in the provided text (only a demo page is mentioned), and the specific hyperparameters for the curriculum learning and meta-token initialization are not fully detailed in the excerpt, which might hinder exact replication.
The paper acknowledges that SynchFormer's attention-switching is not perfectly reliable, occasionally drifting or oscillating, which the semantic-bundled transcription aims to mitigate but may not fully resolve in all edge cases (e.g., heavy occlusion or off-screen speech). The reliance on MLLMs (Gemini 2.5 Pro) for generating the semantic-bundled transcriptions introduces a dependency on external models and potential errors in annotation, although manual verification is claimed. The joint generation model, while competitive, may still suffer from subtle acoustic incoherence compared to specialized single-task models, as indicated by some metric gaps. The benchmarks, while novel, are limited in size (139 and 562 samples respectively), which may not fully capture the diversity of in-the-wild scenarios.
CineDub has significant potential for multimedia production, enabling scalable and realistic video dubbing for movies, TV shows, and online content, particularly in multi-speaker settings. The release of new benchmarks will facilitate further research in this area. However, the technology also raises concerns about deepfake creation and misuse in generating deceptive audio-visual content. The authors' emphasis on realistic evaluation and the complexity of the pipeline may act as a slight barrier to malicious use, but the dual-use nature of such generative models remains a concern. CineDub presents a robust and innovative approach to multi-speaker video dubbing by effectively decoupling holistic visual conditioning from semantic transcription, achieving state-of-the-art results and providing valuable new benchmarks for the community.
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.
This paper provides a critical diagnostic analysis of benchmark contamination in Automatic Speech Recognition models, introducing novel behavioral probes to quantify how models prioritize reference transcripts over acoustic evidence. The work is highly relevant to the current landscape of LLM-audio integration and evaluation, as it addresses a fundamental validity issue in model assessment. The technical contribution is strong, offering mechanistic insights and causal manipulation techniques that advance the field's understanding of model generalization versus benchmark overfitting. While not a new generative architecture or a primary quality metric, it serves as a high-value evaluation study that will likely influence future benchmark design and model training practices.
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Primary: Imperial College London
All Institutions: Imperial College London, University of Surrey
[One sentence main contribution]. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
The paper proposes NAPE, a self-supervised learning framework for audio that adapts the "next-token prediction" paradigm from Large Language Models (LLMs) to continuous audio patch embeddings. The core innovation is a minimalist causal Transformer that predicts the next patch embedding in a log-mel spectrogram sequence using only causal masking and a stop-gradient operator on the target, avoiding reconstruction decoders, tokenizers, or student-teacher setups. The methodology is theoretically sound and aligns with recent trends in joint-embedding predictive architectures (like JEPA) but applies them to the temporal domain of audio. The design choices regarding scanning orders (Raster, Time-major, Zigzag, Diagonal) for the 2D spectrogram grid are a significant technical contribution, as they address the inductive bias required for causal prediction in non-isotropic data. The use of negative cosine similarity with stop-gradient is a known technique (from SimSiam/JSSE), so the novelty lies primarily in the application and specific architectural adaptations (scanning orders, causal masking) rather than a fundamentally new loss function.
The experimental evaluation is comprehensive, covering six standard audio and speech benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP). The authors demonstrate state-of-the-art or competitive performance on several tasks, particularly in fine-tuning scenarios. They also provide linear-probing results, which is crucial for assessing the quality of the learned representations independent of the downstream task head. The scaling analysis (Small, Base, Large) shows consistent performance improvements, suggesting the method is scalable. The ablation studies on scanning orders and predictor heads provide valuable insights into the design space. However, the paper lacks direct comparison with other autoregressive audio models (if any exist in the immediate literature) or joint-embedding methods specifically tailored for audio (like WavJEP), which would strengthen the claim of being a "markedly different" and superior philosophy. The results are strong, but the "state-of-the-art" claim should be contextualized against the specific baselines provided.
The paper provides a GitHub repository link and details the architectural components (Conv2d patch embedding, RoPE, LayerScale, etc.). The training objective is clearly defined with pseudocode. The pre-training data (AudioSet) and preprocessing steps (16kHz, 10s clips, 128 mel bands) are standard and well-documented. The inclusion of ablation studies on key hyperparameters (scanning order, predictor head) enhances reproducibility. The code availability is explicitly stated.
The paper does not explicitly discuss the computational cost of the causal masking compared to masked modeling or contrastive learning. Causal Transformers can be more memory-intensive during training due to the attention mask. The paper also does not explore the robustness of the scanning order across different audio domains (e.g., music vs. speech vs. environmental sounds) in depth, although it mentions it. The reliance on AudioSet for pre-training might limit generalization to out-of-distribution domains not well-represented in AudioSet.
This work contributes to the democratization of audio representation learning by providing a simpler, more unified framework that aligns with the dominant paradigm in NLP and Vision. This could lead to more efficient pre-training pipelines and better transfer learning capabilities for audio tasks. It also opens up new avenues for research into autoregressive audio generation and understanding by establishing a strong baseline for next-embedding prediction. [One sentence main contribution]. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.
Primary: ETH Zurich
All Institutions: ETH Zurich
The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
The paper proposes "Geometric Iterative Retrieval" (GIR), a novel framework for neural audio codec resynthesis. It correctly identifies a gap in the design space between discrete token prediction and continuous regression, positioning GIR in the "continuous, iterative" cell, distinct from diffusion methods by aligning its iterative steps with the RVQ layer hierarchy rather than a noise schedule. The methodology is technically sound: it uses a DeBERTa-v3 backbone to predict continuous latent vectors for each RVQ layer, supervised by a CLIP-style contrastive loss. A key innovation is the replacement of the standard additive RVQ decoding with a self-attention aggregator, allowing for non-additive combination of layer embeddings. This approach effectively leverages the geometric structure of the codebook space, addressing the "geometry-blind" nature of cross-entropy loss in token prediction and the "mean-seeking" behavior of MSE regression. The theoretical framing of the design space is clear and compelling.
The experimental setup is rigorous, evaluating on three diverse datasets (MTG-Jamendo, Common Voice, FMA) using the DAC codec. The baselines are comprehensive, including naive decoding, one-step regression (MSE and Cosine), discrete token prediction (CE), and ablations of the proposed method. The results show that GIR achieves the best Log-Spectral Distance (LSD) and is preferred in double-blind human listening tests over all baselines, including the strong one-step regression baseline. The layer-progression analysis reveals an interesting tension: objective metrics peak at K=3 layers, while human listeners prefer the full K=9 decode, suggesting current objective metrics may not fully capture perceptual quality for residual layers. This finding is valuable for the community. The ablation studies effectively isolate the contributions of the per-layer target, contrastive loss, and attention aggregator.
The paper provides sufficient implementation details for reproduction, including the backbone architecture (DeBERTa-v3), training hyperparameters (AdamW, learning rate schedule, batch size), and dataset mixtures. The code repository is linked. The use of standard codecs (DAC) and metrics (LSD, SI-SDR, FAD) further aids reproducibility.
The method is currently evaluated only on the DAC codec; generalization to other RVQ-based codecs (like EnCodec or SoundStream) is not demonstrated, though the authors claim it is codec-agnostic. The layer-progression analysis shows that adding more layers beyond K=3 degrades objective metrics, which contradicts human preference, indicating a potential misalignment in evaluation metrics or a limitation in how the model handles deep residual layers. The computational cost of the iterative inference (D-1 steps) is higher than one-step regression, though comparable to discrete token prediction.
This work significantly advances the field of neural audio generation by providing a high-fidelity resynthesis method that bridges the gap between discrete and continuous representations. High-quality resynthesis is critical for the fidelity of end-to-end audio generation systems. The insight that codec hierarchy can guide iterative refinement offers a new paradigm for designing generative models in other domains with hierarchical latent structures. The finding regarding the disconnect between spectral metrics and human preference has broader implications for audio evaluation benchmarking. The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institution, Yinwang Intelligent Technology Co., Ltd
VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
The paper proposes VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. The methodology addresses the misalignment of existing metric-based rewards by introducing a human-preference dataset (VAPref-10K) and a three-stage training pipeline: easy cold-start SFT, hard preference alignment via rejection sampling, and dimension-wise reinforcement learning (GRPO). The approach is technically sound, leveraging recent advances in LLM-based reward modeling (CoT, GRPO) and adapting them to the multimodal video-audio domain. The introduction of dimension-wise rewards to provide denser signals than binary labels is a logical and valuable contribution to the field of preference optimization.
The experimental evaluation is comprehensive. The authors construct a new benchmark (VA-Judger-Bench) with in-domain and out-of-domain splits. They demonstrate that VA-Judger outperforms standard metrics (VideoAlign, AudioBox, CLIP, etc.) in predicting human preferences. Crucially, they show that using VA-Judger for post-training (RLHF-style) the LTX-2 model leads to significant improvements in both automatic metrics (JavisBench) and human preference scores (62.3% win rate). The ablation studies on the training stages are well-presented.
The paper provides detailed implementation details, including hyperparameters, model architectures (Qwen3-Omni backbone), and training configurations (DeepSpeed ZeRO 3, BF16). The code repository is linked. The dataset construction process is described, though the specific human annotation protocol details are somewhat condensed. Overall, reproducibility is high.
The reliance on Gemini 3.1 Pro for generating initial CoT responses and the rejection sampling step introduces a dependency on a specific, powerful proprietary model, which may limit the accessibility of the data generation pipeline for some researchers. Additionally, the dataset size (10.3K pairs) is relatively small compared to large-scale web datasets, though sufficient for preference modeling. The generalization to closed-source models (Sora, Veo) is promising but relies on the availability of their outputs for evaluation.
This work significantly advances the field of multimodal generation by providing a robust, human-aligned reward signal for joint video-audio generation. It addresses the critical "reward hacking" problem inherent in metric-based optimization. The dataset and benchmark will serve as valuable resources for future research in multimodal alignment and evaluation. VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.
Primary: Xiaohongshu
All Institutions: Xiaohongshu
[This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
The paper proposes FireRedTTS3, a framework for continuous autoregressive text-to-speech (TTS) that addresses error accumulation through semantically enriched representations. The core innovation is the RedAE tokenizer, which uses a frozen, pretrained Audio Encoder as a semantic teacher to regularize the latent space during training. This approach aims to improve text-speech alignment and stability without adding complex architectural components or multi-stage training pipelines. The generation backbone utilizes an LLM-DiT (Diffusion Transformer) architecture, leveraging the instruction-following capabilities of large language models (Qwen3) for both zero-shot cloning (Base variant) and instruction-controlled editing/design (Instruct variant). The methodology is sound and addresses a known pain point in continuous autoregressive TTS (error accumulation), offering a relatively simple solution compared to prior works that require additional semantic modules or complex tokenizers.
The authors evaluate FireRedTTS3 on four benchmarks: Seed-TTS-Eval, MiniMax-MLS-Test, InstructTTSEval, and Ming-Freeform-Audio-Edit. The results claim state-of-the-art performance in terms of speech intelligibility, speaker similarity, and instruction-following capabilities. The evaluation covers multilingual zero-shot cloning, instruction-controlled voice design, and speech editing. The use of established benchmarks adds credibility. However, the reliance on automated metrics (WER/CER, WavLM similarity, LLM-as-a-judge) for instruction following introduces potential bias, although this is standard in the field. The claim of "best average" performance is supported by the provided tables (referenced as [REF]), suggesting strong empirical results.
The paper provides a GitHub link for code and models. The methodology describes the architecture (Qwen3-style Transformers, DiT, RedAE) and training objectives (GAN, flow-matching, semantic distillation). The use of pretrained components (Qwen3, CAM++, FireRedAudio) aids reproducibility. However, the specific details of the "FireRedAudio" model and the exact composition of the 500k hours of training data are not fully detailed, which may hinder exact replication. The description of the Aggregator and DiT modules is sufficiently detailed for implementation by researchers familiar with these architectures.
The paper acknowledges that Cantonese error rates are high due to the recognition model (Whisper-large-v3) limitations, not necessarily synthesis quality. The reliance on a frozen semantic teacher requires the existence of a high-quality pretrained audio understanding model, which may not be available for all languages or domains. The "simple" architecture still relies on large pretrained LLMs and DiTs, implying significant computational resources for training and inference. The generalization to unseen languages (Portuguese, Ukrainian) is claimed but not deeply analyzed regarding the source of this generalization (linguistic universality vs. acoustic similarity).
FireRedTTS3 contributes to the advancement of controllable and editable speech synthesis, which has applications in accessibility, entertainment, and content creation. The ability to perform instruction-controlled editing and voice design enhances user interaction with speech technologies. However, the ease of voice cloning and editing raises concerns about deepfakes and misuse for fraudulent activities. The open-source release of code and models necessitates careful consideration of safety guardrails and usage policies. [This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
Large Audio Language Models (LALMs) have made rapid progress on standardized benchmarks, yet their deployment in practical media workflows, curation, archival indexing, and content distribution remains largely unrealized. We identify automated audio chapterization, the task of segmenting continuous audio streams into thematically coherent chapters, as a demanding and commercially consequential setting that exposes this gap. Chapterization is challenging because boundaries are defined less by objective acoustic events than by subjective editorial judgment, requiring models to reason sequentially over long acoustic contexts and approximate creator-authored boundary decisions. We present AudioChaps, a post-training framework for aligning end-to-end LALMs for this task via Group Relative Policy Optimization (GRPO) guided by Chain-of-Thought (CoT) reasoning. To support training and evaluation, we curate three datasets: AudioChaps-Alignment, derived from creator-annotated chapter boundaries on YouTube; AudioChaps-CoT, which provides structured supervision for well-formatted, high-quality, and evidence-grounded boundary reasoning; and AudioChaps-Eval, a held-out benchmark for audio chapterization. Applying GRPO directly without a Supervised Fine-Tuning (SFT) cold start, AudioChaps-R1-Zero already improves average F1 by 33 points over the state-of-the-art LALM Audio-Flamingo-3-Think. The AudioChaps framework produces our final aligned LALM, AudioChaps-R1, which improves average F1 by 49 points. These results demonstrate that GRPO-trained LALMs can reliably transform unstructured auditory streams into navigable, structured media. Our code, models, and dataset resources will be released upon acceptance at https://github.com/ta012/AudioChaps.
Primary: University of Surrey
All Institutions: University of Surrey, Huawei Noah's Ark Lab
The paper presents a well-executed study on aligning LALMs for audio chapterization using a novel synthetic CoT dataset and GRPO, demonstrating significant performance gains and contributing valuable resources to the field.
The paper proposes AudioChaps, a framework for aligning Large Audio Language Models (LALMs) for automated audio chapterization. The core methodological contribution is a two-stage post-training pipeline: (1) Supervised Fine-Tuning (SFT) on a synthetic Chain-of-Thought (CoT) dataset (AudioChaps-CoT) to establish a structured reasoning format, and (2) Group Relative Policy Optimization (GRPO) to calibrate boundary decisions against creator-authored annotations. The construction of the CoT dataset via an "audio-to-text modality bridge" using a stronger model (Step-Audio-R1) to generate perception logs and reasoning traces is a notable engineering approach to address the lack of reasoning data for subjective audio tasks. The use of GRPO with rule-based rewards (format and accuracy) is a standard application of recent RL techniques but applied effectively to the audio domain. The approach is sound and addresses a specific gap in LALM capabilities (subjective segmentation vs. objective event detection).
The experimental setup is robust. The authors curate three datasets (Alignment, CoT, Eval) stratified across four acoustic regimes (structured speech, dynamic media, gaming, music). They evaluate on both clip-level binary classification and full-length recording chapter detection. The results show significant improvements over the base model (AF3-Think-8B) and a larger baseline (Step-Audio-R1-32B). The ablation studies effectively isolate the contributions of SFT and GRPO, demonstrating that SFT improves recall while GRPO improves precision. The comparison with an ASR-LLM cascade highlights the advantage of end-to-end audio processing for non-speech cues. The inclusion of human evaluation for reasoning quality adds value. The statistical significance testing via bootstrap is a good practice.
The paper provides detailed descriptions of the dataset construction, training prompts, and hyperparameters. The code and models are promised upon acceptance. The use of standard open-source models (AF3-Think, MOSS-Think) as backbones enhances reproducibility. The synthetic data generation pipeline is described clearly enough to be replicated.
The primary limitation is the reliance on synthetic CoT data, which may contain biases or errors from the teacher model. The task is formulated as a binary boundary detection within a window, which may not capture global context as effectively as native long-context models. The evaluation is limited to YouTube content, which may not generalize to all audio types (e.g., pure music without speech, highly noisy environments). The "creator-annotated" ground truth is a proxy for editorial judgment and may vary in quality or consistency.
This work contributes to the practical deployment of LALMs in media workflows, enabling better content navigation and archival indexing. It demonstrates the potential of RL alignment for subjective audio understanding tasks. The released datasets and benchmarks will facilitate further research in audio chapterization and reasoning. The paper presents a well-executed study on aligning LALMs for audio chapterization using a novel synthetic CoT dataset and GRPO, demonstrating significant performance gains and contributing valuable resources to the field.
Semantic speech tokens should preserve linguistic content while suppressing speaker- and duration-dependent variation inherited from acoustic inputs. We propose Iterative Semantic Token Purification (ISTP), an alternating speech-to-unit (S2U) and text-to-unit (T2U) training procedure guided by text predictability. Starting from an initial S2U tokenizer, each iteration trains a T2U model on its deduplicated token sequences. The decoded T2U predictions then serve as connectionist temporal classification targets for a newly initialized S2U model, whose outputs supervise the next T2U model. This cycle progressively aligns the two token generators and biases the token space toward information recoverable from text. Experiments on Mandarin and English show substantially improved S2U--T2U agreement. Independently trained de-tokenizers further show that the refined S2U and T2U tokens retain sufficient content for high-intelligibility voice conversion and text-to-speech synthesis. In voice conversion, the generated speaking rate follows the reference more closely. The refined tokens also exhibit substantially improved cross-speaker consistency and reduced probe-recoverable speaker information.
Primary: City University of Hong Kong
All Institutions: City University of Hong Kong, Huawei Leibniz Research Center, Chinese University of Hong Kong
[One sentence main contribution]. The paper introduces an iterative refinement procedure that aligns speech and text tokenizers to produce speaker-invariant semantic units. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This research presents a robust method for enhancing the semantic purity of speech tokens by leveraging the inductive bias of text-to-unit models. By iteratively filtering S2U outputs through a T2U model conditioned only on text, the method effectively strips away speaker and duration variations, resulting in tokens that are more consistent across speakers and languages. The comprehensive evaluation, including independent de-tokenizer testing and speaker probing, strongly validates the approach's effectiveness in balancing content preservation with speaker invariance. This contributes meaningfully to the field of speech tokenization by offering a practical solution to the long-standing challenge of acoustic variability in semantic representations.
The paper proposes Iterative Semantic Token Purification (ISTP), a method to align speech-to-unit (S2U) and text-to-unit (T2U) tokenizers. The core innovation is an alternating training loop: an initial S2U tokenizer generates tokens, which are deduplicated and used to train a T2U model. The T2U model then generates pseudo-targets from text, which are used to train a *newly initialized* S2U model via CTC. This cycle repeats, theoretically forcing the S2U tokenizer to discard speaker-specific and duration-dependent variations that are not recoverable from text. The methodology is logically sound and addresses a genuine problem in speech tokenization: the mismatch between acoustic variability and linguistic invariance. However, the approach is essentially a sophisticated form of self-distillation or knowledge transfer using a text-conditioned model as a "filter." While effective, the conceptual novelty is moderate as it relies on established components (HuBERT, FSQ, BART, CTC) in a novel iterative arrangement.
The experiments are comprehensive and well-designed. The authors evaluate on both Mandarin and English, using multiple metrics: S2U-T2U agreement (WER/BLEU), generation utility (TTS/VC WER/SIM), cross-speaker consistency (UED, SelfBLEU-4), and speaker privacy (probe accuracy). The results show significant improvements in consistency and speaker invariance without degrading content preservation. The use of independent de-tokenizers for evaluation is a strong methodological choice that isolates token quality from decoder bias. The inclusion of speaking rate analysis provides additional insight into duration normalization. The baselines are relevant (StableToken, R-Spin, FlexiCodec, etc.). The results are convincing and support the claims.
The paper provides sufficient detail regarding the architecture (HuBERT, FSQ, BART, CTC) and training data sources (LibriSpeech, GigaSpeech, AISHELL-2, etc.). The iterative procedure is clearly defined. However, specific hyperparameters for the BART model, learning rates, and the exact deduplication logic (beyond "consecutive repetitions") are not fully detailed in the text provided. The claim of "8,000 hours" of training data is substantial, but the reproducibility relies on the availability of the specific SOTA components (DSA-Tokenizer, CA-F5-TTS) and the precise implementation of the iterative loop.
The paper does not explicitly discuss the computational cost of the iterative process, which involves training multiple models from scratch in each iteration. This could be a significant barrier to adoption. Additionally, the method relies on the quality of the initial T2U model; if the initial S2U tokens are too noisy or misaligned, the T2U model might learn incorrect mappings. The paper assumes that "text-predictable" content is the only desirable content, which might discard useful prosodic or emotional cues that are not strictly phonetic but are recoverable from context in some T2U models.
This work has significant implications for privacy-preserving speech technologies and efficient speech generation. By reducing speaker information in tokens, it mitigates privacy risks. The improved consistency aids in building more robust multilingual speech systems. The potential for misuse in creating deepfakes with high linguistic fidelity but anonymized voices is a minor concern, but the primary impact is positive for privacy and efficiency. [One sentence main contribution]. The paper introduces an iterative refinement procedure that aligns speech and text tokenizers to produce speaker-invariant semantic units. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This research presents a robust method for enhancing the semantic purity of speech tokens by leveraging the inductive bias of text-to-unit models. By iteratively filtering S2U outputs through a T2U model conditioned only on text, the method effectively strips away speaker and duration variations, resulting in tokens that are more consistent across speakers and languages. The comprehensive evaluation, including independent de-tokenizer testing and speaker probing, strongly validates the approach's effectiveness in balancing content preservation with speaker invariance. This contributes meaningfully to the field of speech tokenization by offering a practical solution to the long-standing challenge of acoustic variability in semantic representations.
Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or decide when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap). The framework uses multi-turn interaction between a Composer and an Instruct model to form a closed evidence-acquisition loop. A Captioner first produces an initial description. Conditioned on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. For credit assignment in variable-length interactions, LOOP-GRPO replaces the trajectory-wide scalar advantage with span-aligned signals: leave-one-out contributions of individual questions to the accumulated evidence, a quality-cost utility for stopping, and an evidence-preservation utility for final synthesis. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the roles to co-evolve. ACE-Cap thus turns captioning from passive one-shot generation into an adaptive process that learns what evidence to acquire, when to stop, and how to preserve it in a long-paragraph caption.
Primary: Kling Team
All Institutions: Kling Team
ACE-Cap introduces a novel agentic framework for fine-grained audio captioning that leverages active evidence acquisition and a specialized reinforcement learning algorithm (LOOP-GRPO) to improve factual accuracy and detail recovery, representing a significant step forward in adaptive audio understanding systems.
The paper proposes ACE-Cap, a framework that reformulates long-paragraph fine-grained audio captioning as an active evidence acquisition problem. The core innovation lies in the agentic loop involving a text-only Composer and an audio-conditioned Instruct model. The Composer identifies missing acoustic details by querying the Instruct model, which provides grounded answers. This addresses the limitation of passive one-shot captioners. The training methodology introduces LOOP-GRPO (Leave-One-Out Per-turn GRPO), which assigns credit to individual questions based on their counterfactual contribution to the final evidence state, rather than using a trajectory-wide scalar advantage. This is a significant methodological contribution to reinforcement learning for agentic systems, specifically addressing the credit assignment problem in variable-length interaction trajectories. The use of a frozen caption-only judge and gold-grounded multiple-choice questions for reward modeling is a practical approach to aligning the agent with fine-grained factual accuracy.
The experimental evaluation is comprehensive, covering multiple benchmarks (Omni-Cloze, MMAR, MMAU, MMSU). The paper demonstrates that ACE-Cap outperforms strong open-source baselines and remains competitive with proprietary models like GPT-4o Audio and Gemini 2.5 Pro. The ablation studies effectively isolate the contributions of the different training stages (Captioner GRPO, Instruct GRPO, Composer LOOP-GRPO, and alternating co-evolution), showing monotonic improvement. The analysis of the Composer's query behavior and the impact of query budget provides valuable insights into the agent's decision-making process. The results support the claim that active evidence acquisition improves fine-grained captioning accuracy.
The paper provides detailed descriptions of the methodology, including the reward function, the LOOP-GRPO algorithm, and the training pipeline. The use of publicly available models (Qwen2.5-Omni-7B, Qwen3-8B) and datasets (ASID-1M) enhances reproducibility. However, the reliance on a proprietary model (Gemini 3.1 Pro) for generating gold captions and MCQs during the offline stage may introduce some variability, although the core training and evaluation are performed on open-source components. The code is not explicitly mentioned as open-sourced in the text, which is a minor drawback for immediate reproducibility.
The paper acknowledges limitations such as the restriction to audio-only inputs and the fixed caption-as-evaluation protocol. The reliance on a fixed question bank derived from gold captions means the reward signal cannot detect unsupported claims outside the predefined set. Additionally, the alternating optimization strategy, while stabilizing training, may not fully capture the complex interdependencies between the Composer and Instruct models compared to joint optimization, although the authors argue it prevents instability. The performance gap with the strongest proprietary models (e.g., Gemini 3.1 Pro on MMAR) suggests there is still room for improvement in handling very complex audio scenes.
This work contributes to the field of audio-language models by introducing an agentic paradigm for fine-grained understanding. The techniques developed for active evidence acquisition and credit assignment in multi-turn interactions have broader applications in other domains requiring iterative reasoning and information gathering, such as visual question answering or document QA. The emphasis on factual accuracy and evidence preservation aligns with the growing need for reliable and interpretable AI systems. ACE-Cap introduces a novel agentic framework for fine-grained audio captioning that leverages active evidence acquisition and a specialized reinforcement learning algorithm (LOOP-GRPO) to improve factual accuracy and detail recovery, representing a significant step forward in adaptive audio understanding systems.
Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmark for instruction-aware speech retrieval, in which natural-language instructions dynamically specify relevance criteria, including semantic content, speaker identity, speaking style, environmental sounds, and their combinations. We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models. Our results reveal that no current method robustly handles all retrieval intents. Text-based approaches perform relatively better at semantic retrieval but struggle with paralinguistic attributes, while speech-based models are moderately better at capturing acoustic properties but falter at following instructions. These findings highlight the need for unified architectures capable of instruction-aware speech retrieval.
Primary: National Taiwan University
All Institutions: National Taiwan University, NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
INSPIRE is the first benchmark for instruction-aware speech retrieval, systematically evaluating the ability of various models to follow natural-language instructions for retrieving speech based on semantic, speaker, style, and environmental attributes, revealing a significant gap in current methods' ability to handle multi-attribute, paralinguistic retrieval.
The paper proposes INSPIRE, a benchmark for instruction-aware speech retrieval. The methodology involves constructing four subsets (DailyTalk, VCTK, Expresso, Synthetic) to test retrieval across semantic, speaker, style, and environmental attributes. The core technical contribution is the formalization of the problem and the construction of the dataset, particularly the synthetic subset which uses GPT-4o-mini for TTS and GPT-5.2 for instruction generation. The evaluation methodology involves testing four distinct retrieval paradigms: Large Audio-Language Models (LALMs), Cascaded Pipelines (ASR/Captioning + Text Retrieval), Self-Supervised Speech Models, and Contrastive Audio-Language Models. The approach is sound in its comparative design, aiming to highlight the gaps in current models. However, the novelty is somewhat limited by the fact that it is primarily a benchmarking paper rather than proposing a new architecture or algorithm. The use of proprietary models (GPT-4o-mini, GPT-5.2) for data generation raises questions about the independence of the benchmark from specific model biases, although this is common in modern AI research.
The experimental evaluation is comprehensive in scope, covering a wide range of existing models and retrieval strategies. The results clearly demonstrate the "modality specialization gap": text-based methods excel at semantic retrieval but fail at paralinguistic tasks, while speech-based models do the opposite. The inclusion of ablation studies on instruction types, pooling strategies, and layer-wise analysis adds depth. The use of Recall@K and NDCG is standard and appropriate. The results are robustly presented with clear tables and radar plots. The finding that no current method handles all intents robustly is significant and well-supported by the data. The synthetic subset's quality assessment (WER, speaker accuracy, emotion accuracy) provides confidence in the dataset's validity.
The paper provides a GitHub link to the code and data, which is a strong point for reproducibility. The dataset construction process is described in detail, including the sources (DailyTalk, VCTK, Expresso) and the generation pipeline for the synthetic subset. The baseline implementations are standard and well-documented. The use of open-source models for most baselines (HuBERT, WavLM, CLAP, E5-Mistral, etc.) ensures that other researchers can reproduce the experiments. The reliance on proprietary models for some baselines (Gemini, GPT-4o-mini) limits full reproducibility of those specific comparisons but does not invalidate the overall benchmark.
A key limitation is the reliance on synthetic data for the most complex multi-attribute tasks. While the synthetic data is high-quality, it may not fully capture the nuances and complexities of real-world speech, such as overlapping speakers, extreme noise, or diverse accents. The instruction generation using GPT-5.2 (a hypothetical or very recent model name, possibly a typo for GPT-4o or similar) introduces a potential bias if the instructions are not diverse enough or if they reflect the specific linguistic patterns of the generator. Additionally, the benchmark is static; it does not address the dynamic nature of user queries in real-world applications. The performance of LALMs is still relatively low, suggesting that the field is in its early stages, which is a limitation of the current state-of-the-art rather than the paper itself, but it highlights the difficulty of the task.
This paper has significant broader impact by defining a new and challenging task in speech processing. It highlights the limitations of current multimodal models and provides a clear direction for future research towards unified, instruction-aware speech retrieval systems. This could lead to more intuitive and powerful speech search engines, accessibility tools, and archival systems. The benchmark will likely spur further research in this area, fostering competition and innovation. The emphasis on paralinguistic attributes (style, emotion, speaker) is particularly important for applications requiring nuanced understanding of human communication. INSPIRE is the first benchmark for instruction-aware speech retrieval, systematically evaluating the ability of various models to follow natural-language instructions for retrieving speech based on semantic, speaker, style, and environmental attributes, revealing a significant gap in current methods' ability to handle multi-attribute, paralinguistic retrieval.
Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is never observed during training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.
Primary: Kuaishou Technology
All Institutions: Kuaishou Technology
SingDance presents a novel and effective framework for compositional zero-shot singing-and-dancing video generation by introducing role-aware audio conditioning and asymmetric staged training, achieving strong performance in motion-beat alignment and lip synchronization while maintaining high visual fidelity.
The paper proposes SingDance, a video diffusion framework that addresses the compositional zero-shot problem of generating singing-and-dancing videos. The core methodological contribution is the formulation of vocal articulation as a semantic "role" (source vs. listener) rather than a simple binary lip-sync instruction. This allows the model to distinguish between a subject producing vocals and a subject listening to off-screen vocals. The architecture utilizes hard-compact routing to select relevant audio tokens (speech from Wav2Vec 2.0, music from MuQ) and injects them via frame-wise joint audio injection into a Video DiT backbone (Wan2.2). A key innovation is the staged training strategy with asymmetric supervision: role control is learned from speech data, and music-conditioned dance is learned from instrumental/dancing-only data, without ever seeing paired singing-and-dancing data during training. The zero-shot composition at inference (Song/Source) combines these separately learned capabilities. The approach is technically sound and elegantly solves the conditioning conflict between speech and music in a unified framework.
The experimental evaluation is comprehensive. The authors curate specific test sets (SingDance-50, Dance-100) to evaluate the proposed capabilities. They compare against strong baselines including MusicInfuser and Wan-S2V. The results demonstrate strong motion-beat alignment and visual fidelity. Crucially, the paired vocal-role switching experiment effectively verifies that the model can switch between singing and listening while maintaining consistent body motion and audio conditioning, a capability that baseline models lacking explicit role conditioning fail to achieve robustly. The lip synchronization metrics are competitive with specialized speech-driven models, despite the parameter efficiency. The ablation studies support the necessity of the music pathway for beat alignment.
The paper provides detailed implementation details, including the backbone (Wan2.2-TI2V-5B), training hardware (64 A100s), hyperparameters (learning rate, batch size), and specific audio encoders (Wav2Vec 2.0, MuQ). The data sources are described (HuMoSet, MA-Data, proprietary collections), though the proprietary nature of some data may limit full reproducibility of the training set. The project page URL is provided, which likely contains code or weights, enhancing reproducibility potential.
The current model is limited to short, single-person clips with a binary, clip-level vocal role. It does not handle within-clip role transitions, duets, or multiple vocal roles. The reliance on hard-compact routing means the model must be explicitly told the role; it does not automatically infer the role from the audio content, which is a design choice but also a limitation for fully autonomous generation. The use of proprietary data for training is a constraint for independent verification.
This work significantly advances the field of multimodal video generation, particularly in the niche of music and speech-conditioned human animation. By enabling compositional zero-shot singing-and-dancing, it lowers the barrier for creating high-quality, personalized performance videos. The role-aware conditioning framework could be extended to other interactive scenarios. However, the ability to generate realistic singing and dancing videos also raises concerns regarding deepfake misuse, although the explicit role conditioning might offer some avenue for detection or control. SingDance presents a novel and effective framework for compositional zero-shot singing-and-dancing video generation by introducing role-aware audio conditioning and asymmetric staged training, achieving strong performance in motion-beat alignment and lip synchronization while maintaining high visual fidelity.
Automatic video dubbing in the wild remains fundamentally limited by two competing constraints: hierarchical methods depend on brittle, multi-stage preprocessing pipelines that severely restrict data scalability and practical deployment, while holistic approaches operating on uncropped video suffer from weak temporal alignment and speaker-utterance ambiguity in multi-speaker settings. To overcome these limitations, we propose CineDub, a unified diffusion-based model that achieves precise multi-speaker dialogue dubbing directly from uncropped videos, without face cropping or speaker diarization. Central to our approach is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm, where holistic visual representations and a semantic-bundled transcription format are encoded independently, yet implicitly coupled through cross-modal training to resolve speaker ambiguity and enable precise multi-speaker multi-turn dialogue dubbing. Building on the unified temporal cues captured by holistic visual features, we further extend CineDub to joint speech and audio generation. We introduce an Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation, and a decoupled textual branch control mechanism to resolve cross-prompt interference during simultaneous generation. We also release two in-the-wild benchmarks, CineDub-Multi for multi-speaker dialogue dubbing and CineDub-SA for video-to-speech-and-audio (V2SA) generation, to enable evaluation under realistic conditions. Experiments show that CineDub achieves state-of-the-art results on established single-speaker dubbing and video-to-audio benchmarks while excelling in multi-speaker dialogue dubbing and acoustically coherent joint generation.
Primary: Monash University
All Institutions: Monash University, University of Chinese Academy of Sciences, Tsinghua University
CineDub presents a robust and innovative approach to multi-speaker video dubbing by effectively decoupling holistic visual conditioning from semantic transcription, achieving state-of-the-art results and providing valuable new benchmarks for the community.
The paper proposes CineDub, a unified diffusion-based framework for end-to-end video dubbing that operates on uncropped videos, addressing the limitations of hierarchical methods (which rely on brittle preprocessing like face cropping and diarization) and holistic methods (which suffer from speaker-utterance ambiguity). The core technical contribution is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm. This involves using SynchFormer features as a holistic visual condition, which the authors argue captures both event-level audio-visual correspondence and fine-grained lip-sync cues through emergent attention mechanisms. To resolve speaker ambiguity, they introduce a "semantic-bundled transcription" format, where speaker descriptions are coupled with transcript segments, encoded by a pre-trained LLM (Gemma-T5). The framework extends to joint speech and audio generation (V2SA), introducing two key mechanisms: Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation by training on audio first, then speech, and a decoupled textual branch control mechanism to prevent cross-prompt interference between speech and audio conditions in the diffusion transformer. The approach is technically sound, leveraging recent advances in diffusion transformers and multimodal alignment, but the novelty lies primarily in the specific conditioning strategies and curriculum learning design rather than fundamental architectural changes.
The authors evaluate CineDub on single-speaker dubbing (GRID, CHEM), multi-speaker dubbing (new CineDub-Multi benchmark), video-to-audio (VGGSound), and joint V2SA (new CineDub-SA benchmark). Results show CineDub outperforms hierarchical baselines (like HPMDubbing, Speak2Dub) and holistic baselines (DeepDubber, DeepAudio) on most metrics, particularly in zero-shot voice cloning and multi-speaker scenarios. The introduction of two new benchmarks, CineDub-Multi and CineDub-SA, is a significant contribution, addressing the lack of realistic, in-the-wild evaluation data for multi-speaker and joint generation tasks. The ablation studies effectively demonstrate the necessity of the semantic-bundled transcription, the ALC curriculum, and the decoupled branches. The use of standard metrics (WER, LSE-D, UTMOS, FDVGG) is appropriate, though the reliance on embedding-based metrics for audio quality (FDVGG, KL) has known limitations regarding perceptual fidelity, which the authors partially address with UTMOS.
The paper provides detailed descriptions of the model architecture, training stages, and data processing pipelines. It mentions the use of pre-trained models (SynchFormer, Gemma-T5, CLIP) and specific datasets (VGGSound, AudioSet, SpeakerVid-5M). The release of the CineDub-Multi and CineDub-SA benchmarks enhances reproducibility for future work. However, the code is not explicitly linked in the provided text (only a demo page is mentioned), and the specific hyperparameters for the curriculum learning and meta-token initialization are not fully detailed in the excerpt, which might hinder exact replication.
The paper acknowledges that SynchFormer's attention-switching is not perfectly reliable, occasionally drifting or oscillating, which the semantic-bundled transcription aims to mitigate but may not fully resolve in all edge cases (e.g., heavy occlusion or off-screen speech). The reliance on MLLMs (Gemini 2.5 Pro) for generating the semantic-bundled transcriptions introduces a dependency on external models and potential errors in annotation, although manual verification is claimed. The joint generation model, while competitive, may still suffer from subtle acoustic incoherence compared to specialized single-task models, as indicated by some metric gaps. The benchmarks, while novel, are limited in size (139 and 562 samples respectively), which may not fully capture the diversity of in-the-wild scenarios.
CineDub has significant potential for multimedia production, enabling scalable and realistic video dubbing for movies, TV shows, and online content, particularly in multi-speaker settings. The release of new benchmarks will facilitate further research in this area. However, the technology also raises concerns about deepfake creation and misuse in generating deceptive audio-visual content. The authors' emphasis on realistic evaluation and the complexity of the pipeline may act as a slight barrier to malicious use, but the dual-use nature of such generative models remains a concern. CineDub presents a robust and innovative approach to multi-speaker video dubbing by effectively decoupling holistic visual conditioning from semantic transcription, achieving state-of-the-art results and providing valuable new benchmarks for the community.
Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.
Primary: University of Edinburgh
All Institutions: University of Edinburgh, Huawei, Wallenberg AI, Autonomous Systems and Software Program (WASP)
This paper presents a robust iterative self-learning framework for expressive TTS that effectively leverages unlabeled data to improve pseudo-label quality and synthesis performance in low-resource settings, demonstrating significant gains in both objective metrics and subjective listener preference.
The paper proposes an Iterative Self-Learning (ISL) framework to address the scarcity of expressive labels in Text-to-Speech (TTS). The core methodological contribution is the integration of "Invert-Classify" into an iterative loop. Invert-Classify is a classifier-free technique that recovers discrete expressive labels (emotion, prominence) by optimizing input embeddings via gradient descent on a frozen generative model's loss function. The authors adapt this for flow-matching models (Matcha-TTS) by fixing stochastic sampling parameters during inversion to ensure convergence. The iterative process involves pseudo-labeling unlabeled data, retraining on the combined dataset, and repeating. The methodology is sound and addresses a specific gap in semi-supervised TTS, moving beyond simple pseudo-labeling to a refinement loop. However, the novelty is somewhat limited by the reliance on existing inversion techniques and the standard IPL protocol, adapted specifically for expressive TTS.
The evaluation is comprehensive, covering two distinct tasks (word-level prominence and utterance-level emotion) and multiple low-resource data splits (0.5% to 20%). The authors employ rigorous metrics including TTSDS2, emotion recognition F1, and subjective A/B listening tests using Bradley-Terry models. The results demonstrate that ISL improves pseudo-label accuracy and downstream synthesis quality compared to single-pass baselines, particularly in low-resource regimes. The analysis of training duration (3 vs 10 vs 50 epochs) provides valuable insight into the stability of the self-learning loop, identifying a "saturation" failure mode with excessive retraining. The use of human listening tests adds significant weight to the claims of perceptual improvement.
The paper provides detailed implementation details, including hyperparameters for optimization, inversion steps, and dataset splits. The use of standard backbones (Matcha-TTS, HiFi-GAN) and public datasets (ESD, Naver-Prosody) enhances reproducibility. The description of the inversion process, including the fixing of timesteps and noise tensors, is sufficiently detailed for replication.
The primary limitation is the reliance on a single backbone architecture (Matcha-TTS), although the authors acknowledge this. The method is sensitive to the amount of seed data; too little prevents bootstrapping, while too much yields diminishing returns. The approach assumes that the generative model can effectively invert to recover labels, which may not hold for all expressive traits or model architectures. Furthermore, the computational cost of iterative gradient-based inversion is non-trivial.
This work contributes to the democratization of expressive TTS by reducing the dependency on expensive, manually annotated expressive datasets. It enables the use of large-scale unlabeled speech corpora for training controllable TTS systems, which has broad implications for accessible speech synthesis, virtual assistants, and content creation. The insights into iterative self-learning dynamics are also applicable to other semi-supervised speech tasks. This paper presents a robust iterative self-learning framework for expressive TTS that effectively leverages unlabeled data to improve pseudo-label quality and synthesis performance in low-resource settings, demonstrating significant gains in both objective metrics and subjective listener preference.
Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
Primary: Kandinsky Lab
All Institutions: Kandinsky Lab
The paper presents a practical and effective method for adding voice cloning to T2AV models with minimal architectural changes, achieving state-of-the-art speaker similarity at the cost of some transcription accuracy.
The paper proposes a method to add voice cloning capabilities to existing Text-to-Audio-Video (T2AV) diffusion models. The core innovation is architectural minimalism: adding a single zero-initialized linear layer to inject a global speaker embedding (via FiLM) and prepending reference audio latents to the input sequence. This allows the model to condition on a reference voice without retraining the entire backbone from scratch. The approach leverages the existing self-attention mechanisms for latent prepending and introduces a new modulation path for the speaker embedding. The methodology is sound and builds logically on existing practices in TTS (speaker embeddings) and diffusion (latent conditioning), but the combination within a T2AV context is a novel application. The zero-initialization ensures stability during the fine-tuning phase.
The evaluation is conducted on a benchmark of 674 speaker-text pairs across 30 speakers from the VCTK corpus. The authors compare their method against five strong baselines, including dedicated TTS models (Qwen3-TTS, XTTS-v2, IndexTTS2) and a joint audio-video model (NAVA). Results show that the proposed method achieves higher speaker-encoder cosine similarity (SECS) across three verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer) compared to baselines. However, this comes at the cost of higher Word Error Rates (WER), indicating a trade-off between speaker fidelity and transcription accuracy. The paper also demonstrates that the audio-only path can be run independently for a ~30x speed-up with minimal loss in speaker similarity. The "no regression" analysis shows that fine-tuning does not degrade the base model's performance on text-to-audio generation.
The paper provides detailed implementation details, including hyperparameters (learning rates, optimizer settings, diffusion steps), architecture dimensions, and training schedules. The use of open-source components (VCTK, Qwen3-TTS encoder, ECAPA-TDNN, etc.) aids reproducibility. However, the base T2AV model (Kandinsky 5.0) is described as having an "internal corpus" and the specific checkpoints are not publicly linked in the text provided, which may hinder exact replication. The code is not explicitly linked in the provided text.
The primary limitation is the trade-off between speaker similarity and speech intelligibility (WER). The model tends to preserve acoustic quirks of the reference, which can lead to hallucinations or errors in the generated text, especially on short or difficult prompts. The performance is also sensitive to reference length and language matching. The method relies on the availability of a pre-trained T2AV model, limiting its applicability to models that have not been trained on such data.
This work enables more personalized and controllable audio-visual generation, which has applications in content creation, dubbing, and virtual avatars. However, the ability to clone voices raises significant ethical concerns regarding misuse for deepfakes or non-consensual voice synthesis. The authors should have included a more robust discussion on these risks and potential mitigation strategies. The paper presents a practical and effective method for adding voice cloning to T2AV models with minimal architectural changes, achieving state-of-the-art speaker similarity at the cost of some transcription accuracy.
Large audio-language models (LALMs) make it possible to interact with language models through speech, music, and environmental sound, but they also introduce a safety surface that is difficult to expose with text-only red-teaming. We study automated audio-grounded red-teaming, where a text query must remain safe in isolation while the joint text-audio input induces harmful target behavior. We propose ARENA, a closed-loop framework that trains a controller on an independent 2,000case text-audio dataset. MD-Judge supplies training rewards and adaptive search feedback, while a separate, non-adaptive Llama Guard 3 evaluator alone labels final outcomes. On 520 held-out AdvBench objectives, ARENA achieves FDR/PSR of 87.9/100.0%, 71.5/96.3%, 68.1/100.0%, and 75.4/98.5% on Audio Flamingo 3, Qwen2-Audio, MiMo-Audio, and GPTAudio, respectively. Ablations show that feedback-based refinement and audio-variant search substantially improve attack discovery.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. [The paper presents ARENA, a novel automated red-teaming framework that uses closed-loop feedback and preference optimization to generate effective audio-grounded jailbreaks for Large Audio-Language Models, demonstrating significant vulnerabilities in current safety mechanisms across multiple state-of-the-art models.]
The paper proposes ARENA, a closed-loop automated red-teaming framework for Large Audio-Language Models (LALMs). The core methodology involves training a controller LLM to generate text-safe queries paired with audio prompts (either speech or environmental sound) that induce harmful compliance in target models. The training utilizes a 2,000-case seed pool, employing Reward-Weighted Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) based on feedback from an MD-Judge model. During inference, the controller iteratively refines prompts based on judge feedback until a success threshold is met. A key technical distinction is the separation of the search feedback mechanism (MD-Judge) from the final evaluation metric (Llama Guard 3), which is intended to prevent overfitting to the judge. The approach addresses the specific challenge of "audio-grounded" jailbreaks where safety is conditional on the audio modality.
The authors evaluate ARENA on four LALMs: Audio Flamingo 3, Qwen2-Audio, MiMo-Audio, and GPT-Audio. They use 520 held-out AdvBench objectives. The results show high Fault Detection Rates (FDR), ranging from 68.1% to 87.9%, significantly outperforming static baselines like AJailBench and JALMBench. The paper includes ablation studies on refinement budgets, target sampling parameters, and audio variant counts, demonstrating that feedback-based refinement and audio variation substantially improve attack discovery. Transferability analysis shows that attacks found on one model can partially transfer to others. The evaluation is comprehensive, covering multiple models and providing detailed failure analysis.
The paper provides a GitHub link for code. The methodology describes the training data construction (2,000 seeds), the reward shaping formulas, and the DPO setup. However, the specific versions of the TTS models (Piper, TangoFlux) and the exact configuration of the MD-Judge and Llama Guard 3 evaluators are not fully detailed in the text provided, which may hinder exact reproduction. The separation of training and evaluation data is clearly stated, which is good practice.
The primary limitation is the reliance on LLM-based judges (MD-Judge and Llama Guard 3) for both training feedback and final evaluation. While the separation is intended to mitigate this, LLM judges are known to have biases and inconsistencies, particularly with audio-grounded reasoning. The paper does not provide human evaluation of the generated jailbreaks or the harmfulness of the responses, which is a significant gap for safety research. Additionally, the effectiveness is limited to the specific audio synthesis models used; different TTS or audio generation models might yield different results. The focus on automated red-teaming also means it may miss nuanced social engineering attacks that require more complex, multi-turn human interaction.
This work has significant implications for the safety of multimodal AI systems. By exposing vulnerabilities in LALMs, it helps developers identify and patch safety gaps before deployment. However, the release of such a powerful red-teaming tool also raises dual-use concerns, as the generated jailbreaks could be misused by malicious actors. The paper responsibly frames this as a safety auditing tool, but the availability of the code and methodology requires careful consideration of access controls. [One sentence main contribution]. [The paper presents ARENA, a novel automated red-teaming framework that uses closed-loop feedback and preference optimization to generate effective audio-grounded jailbreaks for Large Audio-Language Models, demonstrating significant vulnerabilities in current safety mechanisms across multiple state-of-the-art models.]
Audio-Text Foundation Models (ATMs) fail catastrophically under severe acoustic noise, yet existing adaptation strategies either rely on gradient-based Test-Time Adaptation (TTA), which reinforces noise rather than signal, or on prompt tuning that requires privileged noise annotations unavailable at inference. We address these failures with PRISM (Prototype-Rectified Iterative Self-supervised Manifold Denoising), a training-free, source-free TTA framework grounded in the Affine Noise Hypothesis: severe acoustic noise induces a low-rank affine shift in the multimodal latent space, with more than 90% of distortion energy confined to the leading 60 principal components. PRISM estimates and reverses this distortion from an unlabeled target batch using frozen text prototypes as geometric anchors via three closed-form geometric corrections compiled into a single static projection matrix by Affine Bias Regression. At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA while requiring no additional training. On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points, despite never observing its privileged augmented noise prompts. We further identify the Polyphonic Trap, a principled failure mode of subspace deflation for broadband classes, and resolve it via Confidence-Aware Regression (CAR), recovering up to 8.16 percentage points for the worst-affected class.
Primary: Indian Institute of Science Education and Research Bhopal
All Institutions: Indian Institute of Science Education and Research Bhopal, Vellore Institute of Technology Bhopal
This paper presents a novel, geometric approach to test-time adaptation for audio-text models, demonstrating that affine corrections in latent space can effectively mitigate severe acoustic noise without gradients or source data.
The paper proposes PRISM, a training-free, source-free Test-Time Adaptation (TTA) framework for Audio-Text Foundation Models (ATMs). The core theoretical contribution is the "Affine Noise Hypothesis," which posits that severe acoustic noise induces a low-rank affine shift in the multimodal latent space. To correct this, PRISM employs three closed-form geometric operations: Orthogonal Procrustes Cross-modal Alignment (OPCA) to align manifolds, Class-Conditioned Variance Deflation (CCVD) to remove noise-dominant directions via Fisher Linear Discriminant Analysis, and Per-Class Residual Translation. These are compiled into a static projection matrix via Affine Bias Regression (ABR). The approach is mathematically grounded in linear algebra and manifold learning. While the geometric intuition is sound, the novelty lies primarily in the specific combination and application of these techniques to the ATM domain, rather than the introduction of fundamentally new mathematical primitives. The "Polyphonic Trap" analysis is a valuable diagnostic contribution, identifying a specific failure mode where high within-class variance in polyphonic sounds is mistaken for noise.
The evaluation is conducted on UrbanSound8K, ESC-50, and DCASE/TAU 2019 datasets with injected noise. The results show significant improvements over zero-shot baselines and other TTA methods like PCA++ and TDA. Notably, PRISM outperforms the oracle-assisted ContextDA baseline, which is a strong result given that ContextDA has access to privileged noise annotations. The paper provides detailed ablation studies and sensitivity analyses. However, the evaluation relies heavily on synthetic noise injection. While the SNR sweep is thorough, the generalization to real-world, non-stationary acoustic environments (beyond the TAU corpus) is less rigorously demonstrated. The comparison with gradient-based TTA methods highlights the speed advantage but does not fully explore the accuracy-latency trade-off in dynamic streaming scenarios where batch sizes might be smaller than the calibration buffer.
The paper provides detailed algorithmic steps, hyperparameters (K=60, p=0.8, etc.), and implementation details (LAION-CLAP checkpoint). The closed-form nature of the solution enhances reproducibility. However, the code is not publicly linked in the provided text, and the specific prompt templates used for text prototypes are only partially described ("20 diverse prompt templates"). The reliance on a specific foundation model (CLAP) limits direct generalizability to other architectures without adaptation.
The primary limitation is the "Polyphonic Trap," where the method fails for broadband, spectrally dense classes like street music. Although the authors propose Confidence-Aware Regression (CAR) to mitigate this, it adds complexity and the method still struggles with classes where semantic variance overlaps with noise subspace geometry. Additionally, the method assumes the noise distortion is low-rank and affine; if the acoustic environment induces complex, non-linear manifold warping that violates this hypothesis, performance may degrade. The method also requires a calibration batch, which may not be feasible in strictly real-time, single-sample inference scenarios without a warm-up period.
This work contributes to the robustness of audio foundation models in real-world, noisy environments, which is critical for applications like assistive listening, environmental monitoring, and mobile audio search. By providing a computationally efficient, training-free solution, it lowers the barrier for deploying robust ATMs on edge devices. The identification of the Polyphonic Trap offers insights into the limitations of subspace-based denoising for complex audio classes, guiding future research in geometric deep learning for audio. This paper presents a novel, geometric approach to test-time adaptation for audio-text models, demonstrating that affine corrections in latent space can effectively mitigate severe acoustic noise without gradients or source data.
We address the problem of encoding room impulse responses (RIRs) into high-order Ambisonics (HOA) representations from arbitrary and potentially insufficient or incomplete microphone array measurements. This task is fundamentally ill-posed for microphone arrays with limited spatial capture capabilities, such as irregular or sparse arrays, as classical linear methods fail to reconstruct high-order spatial detail. We introduce a diffusion-based generative framework that models the statistical properties of HOA RIRs. This enables device-agnostic encoding from arbitrary microphone arrays, potentially unseen during data measurement. Our approach incorporates a posterior sampling procedure that enforces consistency between the estimated signals and the measurements while plausibly reconstructing spatial information that is unobservable from the limited measurements alone. Experiments on simulated data demonstrate that our method outperforms linear and neural baselines, achieving accurate HOA RIR estimation up to 12th order. A listening test with binaural renderings, including both simulated and measured RIRs, further confirms that the proposed method yields higher perceptual similarity to reference Ambisonics RIRs than all baselines. The flexibility and accuracy of the proposed framework opens new possibilities for scalable acoustics simulations.
Primary: Reality Labs Research, Meta
All Institutions: Reality Labs Research, Meta, Aalto University, Acoustics Lab
This paper presents a significant advancement in spatial audio processing by effectively leveraging diffusion models to solve the ill-posed problem of high-order Ambisonics encoding from sparse measurements, achieving state-of-the-art perceptual and objective performance.
The paper proposes a novel application of diffusion models to the ill-posed inverse problem of encoding Room Impulse Responses (RIRs) into High-Order Ambisonics (HOA) from sparse or irregular microphone arrays. The core methodological contribution is the integration of a device-agnostic diffusion prior with a posterior sampling procedure that enforces data consistency via a range-projected likelihood term. The use of a hybrid time-frequency/time-domain architecture (NCSN++ backbone with a U-Net refinement stage for early reflections) is a well-reasoned design choice given the distinct temporal characteristics of RIRs. The introduction of a compressed spectrogram distance metric for the likelihood guidance is a technically sound innovation to emphasize weak high-order components. The formulation separates the device-agnostic prior from the device-specific likelihood, which is a strong theoretical foundation for generalization.
The experimental evaluation is rigorous and comprehensive. The authors utilize two datasets: a large-scale internal FDTD simulation dataset and the public Treble-10 dataset. They compare against three strong baselines: Linear Least-Squares, a Time-Dependent Neural Encoder, and a Conditional Diffusion model specific to the device. The inclusion of both objective metrics (EDC, NPM) and a subjective listening test (MUSHRA-style with binaural rendering) provides robust validation. The results demonstrate clear superiority in both perceptual similarity and objective metrics, particularly in preserving early reflections and late reverberation tails. The ablation studies effectively isolate the contribution of the range-projection constraint and the hybrid architecture.
The paper provides sufficient detail for reproduction, including dataset descriptions, model architectures (NCSN++ based), training hyperparameters (AdamW, learning rate, batch size), and specific implementation details like the compressed spectrogram definition. The use of standard libraries (PyTorch) and well-known architectures aids reproducibility. However, the reliance on an internal FDTD dataset for the primary training results limits independent verification of the scale of the results, though the public Treble-10 results offer some ground truth.
The primary limitation is the distribution gap between simulated training data and real-world measured RIRs, which the authors acknowledge and demonstrate leads to performance drops on measured data (Eigenmike-64). The method assumes a highly accurate Array Transfer Function (ATF), which may not hold in real-world deployments with calibration errors. Additionally, the iterative sampling process is computationally intensive, making it less suitable for real-time applications without significant acceleration techniques. The evaluation is currently limited to a single device configuration (Aria Glasses) for the device-agnostic claim, though the method is theoretically general.
This work has significant implications for spatial audio processing, particularly for Virtual Reality (VR) and Augmented Reality (AR) applications where scalable and accurate acoustic simulation is crucial. By enabling high-quality HOA encoding from sparse arrays, it lowers the barrier for capturing spatial audio with wearable devices. It also facilitates the generation of large-scale training data for spatial audio models. The device-agnostic nature of the approach promotes interoperability across different hardware platforms. This paper presents a significant advancement in spatial audio processing by effectively leveraging diffusion models to solve the ill-posed problem of high-order Ambisonics encoding from sparse measurements, achieving state-of-the-art perceptual and objective performance.
Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.
Primary: Universitat Pompeu Fabra (Music Technology Group)
All Institutions: Universitat Pompeu Fabra, Music Technology Group
This paper makes a significant contribution to the understanding of representation quality in music foundation models by introducing a systematic layer-wise analysis and a novel pitch-transposition equivariance metric, demonstrating that intrinsic properties can effectively guide layer selection, particularly for tonal tasks where standard metrics fail.
The paper presents a rigorous and systematic layer-wise analysis of 12 music foundation models across three distinct pre-training paradigms (masked, autoregressive, contrastive). The methodology is sound, employing a comprehensive suite of label-free intrinsic metrics (Intrinsic Dimension, Curvature, Anisotropy, Effective Rank, LiDAR, InfoNCE) to characterize representation geometry. The key methodological contribution is the introduction of Pitch-Transposition Equivariance (PTE), a novel metric designed to capture tonal structure that standard geometric metrics miss. The approach of correlating these intrinsic properties with downstream probe performance across 15 diverse MIR tasks provides a robust framework for understanding representation quality without relying solely on task-specific supervision.
The experimental evaluation is extensive, covering a wide range of downstream tasks including tonal, rhythmic, timbral, semantic, and similarity tasks. The results are well-supported, demonstrating that while standard metrics correlate well with non-tonal tasks, they fail for tonal tasks, necessitating the new PTE metric. The paper effectively demonstrates that intrinsic metrics can serve as effective proxies for layer selection, often outperforming trainable multi-layer fusion methods, particularly in low-data regimes. The analysis of depth-wise trends across different model families provides valuable insights into how representation properties evolve.
The paper provides significant detail on the models, datasets (MTG-Jamendo, GiantSteps, NSynth, etc.), and evaluation protocols. The code and extended results are available on the project page, enhancing reproducibility. The use of standard datasets and publicly available models facilitates independent verification.
The study is correlational; it identifies properties associated with good performance but does not establish causality. The metrics are evaluated on frozen representations, so their utility for guiding pre-training or fine-tuning is not directly addressed. The analysis is limited to 12 models, which, while diverse, may not cover all architectural variations. The PTE metric, while promising, is specific to tonal tasks and may not generalize to other musical attributes.
This work provides practical guidelines for selecting layers in music foundation models, potentially reducing the computational cost of model evaluation and deployment. It advances the theoretical understanding of representation quality in audio models, bridging the gap between geometric analysis and practical MIR performance. The findings are relevant to the broader community working with self-supervised audio representations. This paper makes a significant contribution to the understanding of representation quality in music foundation models by introducing a systematic layer-wise analysis and a novel pitch-transposition equivariance metric, demonstrating that intrinsic properties can effectively guide layer selection, particularly for tonal tasks where standard metrics fail.