Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.
Primary: ETH Zurich
All Institutions: ETH Zurich
The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
The paper proposes "Geometric Iterative Retrieval" (GIR), a novel framework for neural audio codec resynthesis. It correctly identifies a gap in the design space between discrete token prediction and continuous regression, positioning GIR in the "continuous, iterative" cell, distinct from diffusion methods by aligning its iterative steps with the RVQ layer hierarchy rather than a noise schedule. The methodology is technically sound: it uses a DeBERTa-v3 backbone to predict continuous latent vectors for each RVQ layer, supervised by a CLIP-style contrastive loss. A key innovation is the replacement of the standard additive RVQ decoding with a self-attention aggregator, allowing for non-additive combination of layer embeddings. This approach effectively leverages the geometric structure of the codebook space, addressing the "geometry-blind" nature of cross-entropy loss in token prediction and the "mean-seeking" behavior of MSE regression. The theoretical framing of the design space is clear and compelling.
The experimental setup is rigorous, evaluating on three diverse datasets (MTG-Jamendo, Common Voice, FMA) using the DAC codec. The baselines are comprehensive, including naive decoding, one-step regression (MSE and Cosine), discrete token prediction (CE), and ablations of the proposed method. The results show that GIR achieves the best Log-Spectral Distance (LSD) and is preferred in double-blind human listening tests over all baselines, including the strong one-step regression baseline. The layer-progression analysis reveals an interesting tension: objective metrics peak at K=3 layers, while human listeners prefer the full K=9 decode, suggesting current objective metrics may not fully capture perceptual quality for residual layers. This finding is valuable for the community. The ablation studies effectively isolate the contributions of the per-layer target, contrastive loss, and attention aggregator.
The paper provides sufficient implementation details for reproduction, including the backbone architecture (DeBERTa-v3), training hyperparameters (AdamW, learning rate schedule, batch size), and dataset mixtures. The code repository is linked. The use of standard codecs (DAC) and metrics (LSD, SI-SDR, FAD) further aids reproducibility.
The method is currently evaluated only on the DAC codec; generalization to other RVQ-based codecs (like EnCodec or SoundStream) is not demonstrated, though the authors claim it is codec-agnostic. The layer-progression analysis shows that adding more layers beyond K=3 degrades objective metrics, which contradicts human preference, indicating a potential misalignment in evaluation metrics or a limitation in how the model handles deep residual layers. The computational cost of the iterative inference (D-1 steps) is higher than one-step regression, though comparable to discrete token prediction.
This work significantly advances the field of neural audio generation by providing a high-fidelity resynthesis method that bridges the gap between discrete and continuous representations. High-quality resynthesis is critical for the fidelity of end-to-end audio generation systems. The insight that codec hierarchy can guide iterative refinement offers a new paradigm for designing generative models in other domains with hierarchical latent structures. The finding regarding the disconnect between spectral metrics and human preference has broader implications for audio evaluation benchmarking. The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.
Primary: University of Surrey
All Institutions: University of Surrey, King's College London, Meta Superintelligence Labs
FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
The paper proposes FlowSep2, a language-queried audio source separation (LASS) system based on Rectified Flow Matching (RFM) with a Diffusion Transformer (DiT) backbone. The core methodological contribution is the integration of "Self-Flow," a self-supervised representation alignment technique adapted from Self-Flow (SRFM), into the generative separation framework. This involves a teacher-student EMA setup where the student predicts a noisy view with mixed flow times, while the teacher provides a cleaner, uniform view, enforcing semantic consistency in the latent space. The approach replaces traditional mask-based discriminative models with a generative paradigm that synthesizes the target source directly from noise, conditioned on mixture audio and text. The use of RFM allows for efficient sampling compared to standard DDPMs. The methodology is technically sound and builds logically on recent advances in flow matching and self-supervised representation learning.
The authors evaluate FlowSep2 on six benchmarks: AudioCaps, VGGSound, ESC-50, MUSDB18, and DCASE 2024 Task 9 (Synth and Real). They employ a comprehensive suite of metrics including FAD, CLAP Score, CLAP$_A$ Score, AudioBox Aesthetics, and SAJ (SAM Audio Judge), supplemented by human subjective evaluations. The results demonstrate state-of-the-art performance on most general LASS benchmarks (AudioCaps, VGGSound, DCASE), significantly outperforming previous discriminative and generative baselines like LASS-Net, AudioSep, and the original FlowSep. Notably, on MUSDB18, FlowSep2 performs competitively in a zero-shot setting despite being trained only on general sound data, whereas SAM-Audio (trained on proprietary music data) leads. The ablation studies effectively isolate the contributions of the DiT backbone, RFM vs. DDPM, and the Self-Flow alignment strategy, showing consistent improvements.
The paper provides detailed descriptions of the model architecture, training hyperparameters (learning rate, batch size, optimizer), and dataset construction. The use of standard components (FLAN-T5, Stable-Audio VAE) aids reproducibility. However, the code repository is not explicitly linked in the provided text (only a demo URL is given), and the exact preprocessing steps for the large-scale training data (5,650 hours) are summarized but not fully detailed in the snippet. The inclusion of ablation studies and specific metric values enhances the ability to verify claims.
A significant limitation is the reliance on generative modeling, which can introduce artifacts or "hallucinations" not present in the target source, although the CLAP$_A$ score suggests good fidelity. The model's performance on music separation (MUSDB18) is inferior to specialized models like SAM-Audio, likely due to the lack of music-specific training data. The computational cost of training and inference for DiT-based flow models is substantial, which may limit accessibility. Additionally, the "Self-Flow" adaptation requires careful tuning of the teacher-student update rates and flow time sampling strategies.
This work advances the field of audio source separation by demonstrating the efficacy of generative flow models for language-guided tasks. It offers a more flexible interface for audio editing and retrieval, allowing users to specify targets via natural language. The improved separation capabilities have applications in multimedia content creation, assistive listening technologies, and audio forensics. The open-source nature of the evaluation (via demo) and the use of public benchmarks contribute to the community's ability to benchmark future systems. FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institution, Yinwang Intelligent Technology Co., Ltd
VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
The paper proposes VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. The methodology addresses the misalignment of existing metric-based rewards by introducing a human-preference dataset (VAPref-10K) and a three-stage training pipeline: easy cold-start SFT, hard preference alignment via rejection sampling, and dimension-wise reinforcement learning (GRPO). The approach is technically sound, leveraging recent advances in LLM-based reward modeling (CoT, GRPO) and adapting them to the multimodal video-audio domain. The introduction of dimension-wise rewards to provide denser signals than binary labels is a logical and valuable contribution to the field of preference optimization.
The experimental evaluation is comprehensive. The authors construct a new benchmark (VA-Judger-Bench) with in-domain and out-of-domain splits. They demonstrate that VA-Judger outperforms standard metrics (VideoAlign, AudioBox, CLIP, etc.) in predicting human preferences. Crucially, they show that using VA-Judger for post-training (RLHF-style) the LTX-2 model leads to significant improvements in both automatic metrics (JavisBench) and human preference scores (62.3% win rate). The ablation studies on the training stages are well-presented.
The paper provides detailed implementation details, including hyperparameters, model architectures (Qwen3-Omni backbone), and training configurations (DeepSpeed ZeRO 3, BF16). The code repository is linked. The dataset construction process is described, though the specific human annotation protocol details are somewhat condensed. Overall, reproducibility is high.
The reliance on Gemini 3.1 Pro for generating initial CoT responses and the rejection sampling step introduces a dependency on a specific, powerful proprietary model, which may limit the accessibility of the data generation pipeline for some researchers. Additionally, the dataset size (10.3K pairs) is relatively small compared to large-scale web datasets, though sufficient for preference modeling. The generalization to closed-source models (Sora, Veo) is promising but relies on the availability of their outputs for evaluation.
This work significantly advances the field of multimodal generation by providing a robust, human-aligned reward signal for joint video-audio generation. It addresses the critical "reward hacking" problem inherent in metric-based optimization. The dataset and benchmark will serve as valuable resources for future research in multimodal alignment and evaluation. VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
We propose $\textit{recursive encoder and decoder}$ (RED) for building a single deep neural network (DNN) model that can separate multi-speaker mixtures containing unknown numbers of speakers and variable numbers of microphones arranged in an unknown geometry, a task that has not been studied yet. The decoder of RED recursively detects whether there are active speakers left and separates one speaker at a time. It is designed to be trained in an end-to-end fashion to improve separation performance. The encoder of RED recursively encodes each microphone channel of the input mixture, sequentially incorporating spatial cues. Combining both, the DNN can be trained to separate mixtures not only with unknown numbers of speakers but also with variable numbers of microphones, achieving state-of-the-art performance on multiple public datasets.
Primary: unknown (Affiliations not explicitly stated in text, authors are Fulin Wu and Zhong-Qiu Wang)
All Institutions: unknown
[One sentence main contribution]. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
The paper proposes REDnet, a recursive encoder-decoder architecture for speech separation. The core novelty lies in handling two variable dimensions simultaneously: the number of microphones (via a recursive encoder that aggregates channel features sequentially) and the number of speakers (via a recursive decoder that peels off one speaker at a time using a gating mechanism and a speaker detector). The methodology is technically sound, building upon established blocks (TF-GridNet) and concepts (Permutation Invariant Training, recursive separation). The recursive encoder design effectively addresses the variable microphone input problem by avoiding fixed-size pooling or explicit pairwise attention, offering a linear complexity solution. The recursive decoder's use of a gating mechanism to update the residual representation is a logical extension of prior recursive separation works, adapted for end-to-end training with specific loss functions to encourage orthogonality between separated speakers and residuals. While the combination is novel, the individual components are incremental improvements on existing paradigms (TAC for variable mics, OR-PIT/SepTDA for unknown speakers).
The experimental evaluation is comprehensive and rigorous. The authors decompose the problem to validate the Recursive Encoder (RE) and Recursive Decoder (RD) separately before presenting the full REDnet. They use standard public datasets (WSJ0, WHAM!, CHiME-4) and compare against strong baselines (TAC, FlexIO, MUSE, SR-CorrNet). The results show state-of-the-art performance on multiple metrics (SI-SDR, SDR, PESQ, STOI) across various configurations (variable mics, unknown speakers, noisy/reverberant conditions). The ablation studies on the RDnet component are particularly helpful in isolating the contributions of the gating mechanism, speaker interaction module, and loss functions. The evaluation on real-recorded signals (DNSMOS) adds practical value. The comparison with FlexIO is fair, noting that FlexIO assumes known speaker counts, whereas REDnet does not, making REDnet's performance gains even more significant.
The paper provides detailed architectural descriptions, including hyperparameters for the TF-GridNet blocks, optimizer settings, and loss weights. The dataset construction and preprocessing steps are clearly defined. The use of standard datasets and metrics facilitates reproduction. However, the specific implementation details of the "modified TF-GridNet blocks" (e.g., exact SwiGLU placement, kernel sizes) are described but might require careful reading to implement exactly. The code is not linked, which is a minor hindrance to immediate reproducibility, but the description is sufficient for a competent researcher.
The primary limitation is that the model's performance depends on the accuracy of the speaker detector in the recursive decoder. If the detector fails to stop (false positive) or stops too early (false negative), performance degrades. The paper reports high counting accuracy, but this is on controlled datasets. In highly noisy or reverberant real-world scenarios, detector robustness might be a challenge. Additionally, the recursive nature of the decoder implies that errors can propagate if the gating mechanism does not perfectly isolate the speaker, although the residual update mechanism mitigates this. The model assumes that the number of microphones is known at inference time (to process the input), which is standard, but the "unknown geometry" aspect is handled by the channel-agnostic aggregation, which is a strength but relies on the assumption that spatial cues are sufficiently captured by the recursive encoding.
This work contributes to the development of more robust and flexible audio processing systems, which are crucial for real-world applications like hearing aids, smart speakers, and teleconferencing systems where the number of speakers and microphones can vary. By unifying these capabilities into a single model, it reduces the need for multiple specialized models and simplifies deployment. The recursive approach is generalizable to other source separation tasks (e.g., music, sound event detection) and potentially to other modalities where variable input/output dimensions are a challenge. [One sentence main contribution]. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
Non-verbal vocalizations (NVs), such as laughter, coughs, and sighs, are essential for expressive TTS, but the effectiveness of preference optimization for NV generation remains poorly understood. We systematically study preference optimization for NV-capable TTS, focusing on preference signals, preference-pair construction, and DPO-based optimization objectives. We formulate an NV-aware character error rate (NV-CER) by treating NV tags as distinct output symbols and computing a weighted pinyin-based CER over both verbal and non-verbal content, enabling controllable optimization of NV realization without modifying the underlying optimization algorithm. Experiments on Emilia-NV and the augmented NV-Bench covering 18 NV types reveal how different design choices affect NV realization and lexical fidelity, and establish an effective setup using standard DPO. Objective, LLM-based, and human evaluations provide converging evidence for our findings, offering practical insights into NV-aware post-training for expressive TTS.
Primary: Nanyang Technological University
All Institutions: Nanyang Technological University, Nanjing University, National University of Singapore, LIGHTSPEED
This paper presents a systematic evaluation of DPO for non-verbal vocalization synthesis in TTS, introducing an NV-aware CER metric and demonstrating that NV-specific preference signals improve NV accuracy and overall synthesis quality over standard baselines.
The paper proposes a systematic study of Direct Preference Optimization (DPO) for Non-Verbal Vocalization (NV) synthesis in Text-to-Speech (TTS) systems. The core methodological contribution is the formulation of an NV-aware Character Error Rate (NV-CER) that treats NV tags as distinct symbols and computes a weighted edit distance over pinyin and NV tokens. This metric is used to construct preference pairs (preferred vs. rejected) for DPO. The authors also introduce a weighting mechanism ($w_{NV}$) to control the trade-off between NV realization and lexical fidelity. While the application of DPO to TTS is not entirely new, its systematic application and evaluation specifically for the nuanced task of NV generation, along with the specific NV-CER metric, provides a structured approach to a previously under-explored area. The methodology is sound but largely incremental, relying on standard DPO formulations and existing ASR models (SenseVoice) for preference signals.
The experiments are conducted on the Emilia-NV dataset and the NV-Bench. The authors evaluate various preference signals (ASR CER, NV-CER, NV-CER+UTMOS) and loss formulations (DPO, DPO+SFT). Results show that NV-CER-based preference signals outperform conventional ASR-CER and baseline SFT models in both objective metrics (NV-CER, PCER, UTMOS, DNSMOS) and LLM-based/human evaluations. The ablation studies on preference pair construction (Synthetic vs. Ground Truth) and NV-weight tuning provide valuable insights. However, the improvements in perceptual quality (UTMOS/DNSMOS) are modest, and the human evaluation shows high tie rates, suggesting that while NV accuracy improves, naturalness gains are less pronounced. The use of LLM-based evaluation adds a modern layer of assessment, though it correlates with objective metrics.
The paper provides sufficient implementation details, including the base model (CosyVoice2-0.5B), training hyperparameters (learning rates, epochs, batch sizes), and the specific NV-ASR model used (SenseVoice-Small fine-tuned). The dataset (Emilia-NV) and benchmark (NV-Bench) are publicly available or can be constructed as described. The code is not explicitly linked, but the methodology is clear enough for reproduction by researchers familiar with LLM-based TTS systems.
The primary limitation is that the preference signal relies entirely on an ASR model, which may not perfectly correlate with human perception of NV naturalness or correctness, especially for ambiguous NVs. The paper acknowledges that the base model rarely omits NVs, making "no-NV" a weak negative example, which limits the diversity of preference pairs. Furthermore, the improvements in perceptual quality are marginal, and the method does not fundamentally change the underlying TTS architecture, relying instead on post-training. The reliance on pinyin for Mandarin also limits immediate generalizability to other languages without adaptation.
This work contributes to the field of expressive TTS by providing practical guidelines for incorporating NVs into preference optimization pipelines. It helps bridge the gap between lexical TTS and more human-like, expressive speech synthesis. The NV-CER metric offers a standardized way to evaluate NV generation, which could facilitate future research in this area. The findings are relevant for developers of conversational AI agents and virtual assistants requiring more natural interaction capabilities. This paper presents a systematic evaluation of DPO for non-verbal vocalization synthesis in TTS, introducing an NV-aware CER metric and demonstrating that NV-specific preference signals improve NV accuracy and overall synthesis quality over standard baselines.
Spatial audio is a key component of immersive $360^\circ$ media, yet high-quality spatial capture remains limited in real-world speech-dominant scenes. We study visually guided First-Order Ambisonics (FOA) speech spatialization in the wild: given aligned $360^\circ$ video and an omnidirectional audio track, we recover the missing directional FOA components. To support this task, we introduce YT-SPEECH, a speech-oriented $360^\circ$ video-FOA dataset curated from YouTube. We propose a two-stage Localizer-Renderer framework, where an audio-visual segmentation backbone provides frame-wise spatial heatmaps and a conditional complex-domain U-Net reconstructs directional FOA signals from the omnidirectional channel. A confidence-based gating strategy stabilizes conditioning under ambiguous acoustic conditions. Experiments show improved reconstruction fidelity, spatial accuracy, and perceptual speech quality relative to ablated variants and prior approaches.
Primary: University of Surrey
All Institutions: University of Surrey
This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
The paper proposes a two-stage framework, Localizer-Renderer, for visually guided First-Order Ambisonics (FOA) speech spatialization. The Localizer utilizes an Audio-Visual Segmentation (AVS) backbone to generate spatial heatmaps from 360-degree video, which are then used as conditional priors for a complex-domain U-Net Renderer. The approach introduces a confidence-based gating mechanism to modulate the conditioning strength based on the entropy and peakiness of the spatial prior. While the integration of AVS for spatial grounding in audio generation is a logical extension of current multimodal trends, the core methodology relies on established components (U-Net, AVS backbones, FOA encoding) rather than introducing fundamentally new architectural primitives. The "confidence gating" is a heuristic modification rather than a deep theoretical innovation. The use of complex-valued networks for phase preservation is a known best practice in this specific subfield, not a novel contribution.
The authors curate a new dataset, YT-SPEECH, which is a valuable resource for the community, addressing the scarcity of paired 360-video and FOA speech data. The experimental evaluation includes ablation studies on the Localizer and Renderer components, as well as comparisons with analytic baselines (AmbiEnc, Pyroomacoustics) and a prior end-to-end model (SAG). The results show improvements in spatial accuracy (DOA error) and perceptual quality (PESQ, MOS). However, the subjective evaluation is limited to a small-scale listening test with only 9 participants, which reduces the statistical power of the perceptual claims. The comparison with SAG is conducted on SAG's own datasets, which is good for compatibility, but the lack of comparison with other recent state-of-the-art video-to-audio models (e.g., ViSAGe, OmniAudio) limits the assessment of relative performance in the broader landscape.
The paper provides sufficient detail on the architecture, loss functions, and training strategy (pretraining on Sphere360, fine-tuning on YT-SPEECH). The dataset curation pipeline is described in detail. However, the code is not explicitly linked in the text (only a demo URL is provided), and the specific hyperparameters for the AVS backbone fine-tuning are not fully enumerated, which may hinder exact reproduction. The reliance on proprietary or hard-to-access components (e.g., specific Whisper versions, YOLOv8-nano configurations) adds minor friction.
The authors acknowledge the limited scale of the dataset (8.9 hours) and reduced stability under overlapping sources and acoustically complex scenes. The method's performance is heavily dependent on the quality of the visual localization; if the AVS model fails to segment the speaker correctly, the spatial prior will be misleading, potentially degrading performance. The small sample size of the subjective listening test is a significant limitation for validating perceptual claims.
This work contributes to the field of immersive media and spatial audio, which has applications in virtual reality, telepresence, and accessible media for visually impaired users. The release of the YT-SPEECH dataset is a positive contribution to the community, enabling further research in visually guided spatial audio. However, the use of YouTube data raises potential copyright and privacy concerns regarding the training data, although the authors note manual inspection for quality and consistency. This paper presents a practical framework for visually guided FOA speech spatialization, leveraging existing AVS models to guide complex-domain audio generation, and contributes a new dataset to the field.
The text contains content addressed to automated reviewers. Its score has been penalised, and the ranking below should not be trusted.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
Text addressed to an automated reviewer was found in this paper. The impact score shown has already been penalised for this, but treat the ranking with suspicion.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
Primary: Xiaohongshu
All Institutions: Xiaohongshu
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
The paper proposes FireRedAudio, a 9B-parameter unified audio-language model that decouples continuous input representations for understanding versus generation. The core methodological contribution is the use of two distinct pathways: a dedicated Audio Encoder (initialized from Whisper-large-v3) for understanding tasks (ASR, audio QA) and a RedAE-based continuous autoencoder pathway for speech generation (TTS, editing). The generation pathway utilizes a Flow-Matching DiT conditioned on LLM hidden states to generate RedAE latents, which are then decoded to waveforms. The authors argue that understanding benefits from compact, semantic features while generation requires reconstructible, fine-grained features, and that fusing them degrades performance. The training strategy involves a five-stage progressive curriculum, starting with adapter alignment, then encoder adaptation, unified mid-training, multitask post-training with CoT, and finally long-context extension. This approach is technically sound and addresses a known tension in unified multimodal models. The use of flow-matching for continuous latent generation is a modern and effective choice compared to standard diffusion or autoregressive discrete token prediction.
The evaluation covers a broad suite of tasks: ASR (LibriSpeech, FLEURS, etc.), Audio Understanding (MMAU, MMSU), Zero-shot TTS (Seed-TTS-Eval), Instruct TTS (InstructTTSEval), and Speech Editing (Ming-Freeform-Audio-Edit). The results are competitive or leading on most benchmarks. Notably, the model achieves strong performance in Instruct TTS instruction following and significant improvements in speech editing over prior work (Ming-UniAudio-Edit). The long-form audio understanding capability (up to 1 hour) with second-level timestamp accuracy is a strong practical contribution. The ablation studies and comparisons are comprehensive. The use of an LLM-as-a-judge for temporal grounding evaluation is a reasonable proxy but introduces potential bias; however, the strict/content metrics provide some robustness.
The paper provides detailed architectural descriptions, training stages, loss functions, and hyperparameters (learning rates, warmup steps). The code is made available on GitHub. The use of open-source components (Qwen3.5-9B, Whisper-large-v3) aids reproducibility. The RedAE model is described in detail, including its hybrid GAN and distillation loss. The main limitation for perfect reproducibility is the specific data mixtures and the proprietary nature of some training data (e.g., specific Instruct TTS datasets), though the sampling strategy is well-documented.
The model relies on a large 9B parameter backbone, which implies significant computational requirements for training and inference compared to smaller specialized models. The decoupled representation, while effective, adds architectural complexity and potentially increases inference latency due to the need to manage two distinct pathways and the DiT generation loop. The performance on very long-form audio (1 hour) might still suffer from context window limitations or memory constraints during inference, although the paper claims success. The reliance on a frozen teacher for RedAE distillation means the final RedAE latents are bounded by the teacher's capabilities.
FireRedAudio contributes to the field of unified multimodal AI, demonstrating that decoupled representations can effectively balance the conflicting objectives of understanding and generation. This has implications for developing more capable and efficient voice assistants, content creation tools, and accessibility technologies. The ability to perform high-quality speech editing and TTS with natural language instructions opens up new creative and practical applications. However, like all generative audio models, it raises concerns about deepfakes and misuse for impersonation, which the authors do not explicitly address in the provided text. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video, environmental sound, music and speech. We organize interaction around camera intent: in first-person scenes, it specifies observer motion, while in third-person scenes, camera--character dynamics are learned from data without view-specific controllers. Discrete commands and continuous poses are mapped to a shared metric-scale relative 6-DoF trajectory, with dataset-level calibration preserving motion magnitude across heterogeneous data. To jointly learn audio-visual generation and trajectory control, we construct a complementary data engine and adopt progressive training followed by autoregressive post-training for long-horizon generation. Extensive evaluations show that \model achieves strong trajectory following and high visual quality on public world-model benchmarks, supporting both first- and third-person interaction across varied subjects, and maintaining synchronized environmental sound and speech over long-horizon generation.
Primary: JD (Joy Future Academy)
All Institutions: Joy Future Academy, JD, Beihang University, Stanford University
EchoWM presents a comprehensive framework for omnimodal world modeling by unifying camera-centric navigation with joint audio-visual generation through a novel data engine and progressive training curriculum. The technical contribution of mapping heterogeneous control signals to a shared metric-scale trajectory and integrating this via UCPE is a significant step towards more controllable and interactive generative media, although the reliance on recovered geometry and the limitation to 720p resolution remain notable constraints.
The paper proposes "EchoWM," an omnimodal world model that jointly generates 720p video, environmental sound, music, and speech conditioned on a continuous 6-DoF camera trajectory. The core methodological contribution lies in the "camera intent" interface, which unifies first-person and third-person navigation by mapping discrete commands and continuous poses to a shared relative trajectory. The authors introduce a "complementary data engine" combining internal gameplay, internet gameplay, UE simulation, and web video, processed through separate audio-visual and geometry paths. A key technical detail is the use of Unified Camera Positional Encoding (UCPE) to inject relative camera geometry into the video transformer's attention mechanism, avoiding the need for separate camera encoders. The training strategy is progressive: Audio-Visual Continued Pretraining (AV-CPT), Action-SFT (freezing the backbone to learn trajectory conditioning), Joint-FT, and Autoregressive Post-Training (using Self-Gradient Forcing) for long-horizon generation. The approach is technically sound, addressing the distributional mismatch between audio-visual quality and control precision by curating specific data mixtures for each training stage.
The paper claims state-of-the-art performance on public benchmarks (WBench Navigation, SANA-WM-Bench). It emphasizes the ability to maintain synchronized audio and video over long horizons. However, the provided text is truncated, limiting the assessment of the full experimental results, ablation studies, and quantitative metrics (e.g., FAD, CLAP score, UTMOS, or human evaluation scores). The reliance on "public world-model benchmarks" suggests a focus on visual and control consistency, but the specific metrics for audio quality and synchronization fidelity are not detailed in the excerpt. The claim of "strong trajectory following" is supported by the controlled data engine, but the evaluation of speech synchronization specifically is less clear from the text.
The paper provides detailed descriptions of the data processing pipeline, including metric pose recovery using ViPE, VGGT-Omega, and MoGe-2. It specifies the use of Qwen3-Omni and Gemini-3-Pro for annotation. The training stages and loss functions are mathematically defined. However, the exact hyperparameters, the specific versions of the base models (e.g., the pretrained joint audio-visual diffusion transformer), and the code for the data engine are not included in the text. The reliance on proprietary internal gameplay data and specific annotation pipelines may hinder exact reproducibility without significant effort.
The model is limited to 720p resolution, which is modest compared to recent high-fidelity video generators. The interaction is restricted to camera/navigation intent; semantic actor actions (attacking, jumping) are not modeled. The audio generation, while synchronized, is conditioned on visual trajectory rather than direct acoustic modeling, which may limit the fidelity of complex acoustic environments. The reliance on recovered camera poses from internet video introduces potential noise, although quality filtering is applied. The truncation of the paper prevents a full assessment of failure modes or edge cases in long-horizon generation.
EchoWM contributes to the field of generative AI by enabling more immersive and interactive multimedia experiences. The ability to generate synchronized audio-visual content in response to navigation could impact gaming, virtual reality, and simulation. However, the generation of realistic speech and environmental sounds raises concerns about deepfakes and misinformation, particularly if the technology is misused to create deceptive audio-visual content. The paper does not explicitly discuss safety mitigations. EchoWM presents a comprehensive framework for omnimodal world modeling by unifying camera-centric navigation with joint audio-visual generation through a novel data engine and progressive training curriculum. The technical contribution of mapping heterogeneous control signals to a shared metric-scale trajectory and integrating this via UCPE is a significant step towards more controllable and interactive generative media, although the reliance on recovered geometry and the limitation to 720p resolution remain notable constraints.
Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.
Primary: LIGHTSPEED
All Institutions: LIGHTSPEED, National University of Singapore, Nanyang Technological University, Independent Researcher
EmoTra-TTS makes a significant contribution to expressive TTS by introducing a novel framework for smooth intra-utterance emotion transitions, combining synthetic data generation with a robust dual-stage conditioning mechanism. The rigorous evaluation and clear ablation studies demonstrate the effectiveness of the proposed methods, although generalization beyond the training dataset remains an area for future work.
The paper addresses the underexplored problem of intra-utterance emotion transitions in TTS, moving beyond static emotion conditioning. The proposed EmoTra-TTS framework is built on three pillars: (1) a synthetic data generation pipeline using multi-pass flow blending to create frame-aligned transition samples, addressing data scarcity; (2) dual-stage conditioning where an LLM handles prosodic planning via temporal VAD tokens, and a flow decoder realizes acoustic features via frame-level VAD embeddings; and (3) a direction-magnitude decoupled injection mechanism using LayerNorm and a fixed scale to prevent content degradation when injecting emotion signals into a frozen decoder. The methodology is sound and addresses specific failure modes of previous approaches (e.g., content collapse, magnitude mismatch) with rigorous ablation studies. The separation of prosodic planning and acoustic realization is a logical extension of current LLM-based TTS architectures.
The experimental evaluation is comprehensive, including MOS-based metrics (Quality, Emotion, Transition), speaker similarity, WER, and pairwise preference tests against four open-source baselines and two commercial systems. The results show significant improvements in transition smoothness (MOS-Tra) and overall preference. The inclusion of an objective metric (Prosodic Jerk Ratio) strengthens the findings. The ablation studies effectively isolate the contributions of each component. However, the evaluation is limited to the EmoVoice-DB dataset, which restricts claims about generalization to other speakers or languages. The use of synthetic data for training, while effective, introduces a domain gap that is partially mitigated by the quality filtering but remains a limitation.
The paper provides detailed descriptions of the architecture, training objectives, and hyperparameters. The code and demo are publicly available. The synthetic data generation pipeline is clearly described, allowing for reproduction of the training data. The use of a frozen backbone (CosyVoice2) and specific injection mechanisms makes the implementation straightforward to replicate.
The primary limitation is the reliance on the EmoVoice-DB dataset, which has limited speaker diversity and language coverage. The paper acknowledges that zero-shot generalization to unseen speakers is not demonstrated. Additionally, the emotion control stability for subtle VAD differences is noted as inconsistent. The piecewise linear interpolation of VAD trajectories is a simplification of natural emotion dynamics.
EmoTra-TTS contributes to the field of affective computing and expressive speech synthesis. By enabling smooth, controllable emotion transitions, it opens up new possibilities for storytelling, dubbing, and interactive virtual agents. The ethical statement addresses potential misuse, such as generating deceptive audio, and emphasizes responsible use. The work encourages further research into dynamic emotion modeling in speech. EmoTra-TTS makes a significant contribution to expressive TTS by introducing a novel framework for smooth intra-utterance emotion transitions, combining synthetic data generation with a robust dual-stage conditioning mechanism. The rigorous evaluation and clear ablation studies demonstrate the effectiveness of the proposed methods, although generalization beyond the training dataset remains an area for future work.
Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues over multiple audio inputs, requiring models to identify degradation types, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and explain low-level acoustic phenomena in natural language. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. MRMAD reveals an important yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
Primary: Northeastern University
All Institutions: Northeastern University, Bose Corporation, Stony Brook University
[One sentence main contribution]. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
The paper introduces MRMAD, a benchmark designed to evaluate Large Audio-Language Models (LALMs) on their ability to perceive and reason about audio degradation. The methodology involves constructing a dataset of 8,400 multiple-choice questions across three domains (speech, music, sound) and nine degradation types (e.g., noise, reverberation, compression). The evaluation is framed as multi-turn dialogues where models must identify degradation types, compare severity between two clips, or rank three clips by severity. The approach is methodologically sound for its purpose: it isolates low-level perceptual capabilities from high-level semantic understanding by using clean references and controlled degradations. The multi-turn design is a clever solution to the problem of comparing multiple audio inputs without temporal localization confusion. However, the novelty is moderate as it is essentially a specialized benchmark addition to the growing list of audio evaluation suites, rather than proposing a new model architecture or fundamental algorithm.
The experimental setup is comprehensive, evaluating 18 representative LALMs, including open-source models (SALMONN, Qwen2-Audio, Audio Flamingo) and closed-source/reasoning models (GPT-4o, Gemini, Qwen3-Omni). The results clearly demonstrate that current state-of-the-art models perform near random chance on degradation type identification and struggle significantly with severity comparison and ranking. This provides a valuable diagnostic baseline. The evaluation covers a wide range of model sizes and types, strengthening the generalizability of the findings. The use of balanced sampling and specific distractor generation for multiple-choice questions adds rigor to the evaluation protocol.
The paper provides a clear data construction pipeline, specifying source datasets (VCTK, LibriSpeech, MUSDB18, etc.), degradation parameters, and question generation templates. The code and dataset are made available via a GitHub link, which significantly enhances reproducibility. The multi-turn format and template-based prompt generation are described in sufficient detail for others to replicate the evaluation.
A key limitation is the reliance on synthetic degradations applied to high-quality source audio. While this allows for controlled severity levels, it may not fully capture the complexity of real-world audio degradation, which often involves complex, non-linear interactions between noise sources and signal characteristics. Additionally, the multiple-choice format, while efficient for large-scale evaluation, may not fully capture the nuanced reasoning capabilities of models compared to open-ended generation tasks. The exclusion of certain degradations for specific domains (e.g., echo in music) is justified but limits the universality of the benchmark. Finally, the evaluation is limited to 18 models; as the field evolves rapidly, the benchmark will need continuous updating.
This work has significant implications for the development of robust audio-language models. By highlighting the gap in low-level audio perception, it directs future research towards improving model robustness to real-world acoustic conditions. This is crucial for applications in accessibility (e.g., audio description for visually impaired users in noisy environments), audio restoration, and high-fidelity communication systems. The benchmark serves as a necessary tool for the community to track progress in this underexplored area. [One sentence main contribution]. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.
Primary: University of Surrey
All Institutions: University of Surrey, King's College London, Meta Superintelligence Labs
FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
The paper proposes FlowSep2, a language-queried audio source separation (LASS) system based on Rectified Flow Matching (RFM) with a Diffusion Transformer (DiT) backbone. The core methodological contribution is the integration of "Self-Flow," a self-supervised representation alignment technique adapted from Self-Flow (SRFM), into the generative separation framework. This involves a teacher-student EMA setup where the student predicts a noisy view with mixed flow times, while the teacher provides a cleaner, uniform view, enforcing semantic consistency in the latent space. The approach replaces traditional mask-based discriminative models with a generative paradigm that synthesizes the target source directly from noise, conditioned on mixture audio and text. The use of RFM allows for efficient sampling compared to standard DDPMs. The methodology is technically sound and builds logically on recent advances in flow matching and self-supervised representation learning.
The authors evaluate FlowSep2 on six benchmarks: AudioCaps, VGGSound, ESC-50, MUSDB18, and DCASE 2024 Task 9 (Synth and Real). They employ a comprehensive suite of metrics including FAD, CLAP Score, CLAP$_A$ Score, AudioBox Aesthetics, and SAJ (SAM Audio Judge), supplemented by human subjective evaluations. The results demonstrate state-of-the-art performance on most general LASS benchmarks (AudioCaps, VGGSound, DCASE), significantly outperforming previous discriminative and generative baselines like LASS-Net, AudioSep, and the original FlowSep. Notably, on MUSDB18, FlowSep2 performs competitively in a zero-shot setting despite being trained only on general sound data, whereas SAM-Audio (trained on proprietary music data) leads. The ablation studies effectively isolate the contributions of the DiT backbone, RFM vs. DDPM, and the Self-Flow alignment strategy, showing consistent improvements.
The paper provides detailed descriptions of the model architecture, training hyperparameters (learning rate, batch size, optimizer), and dataset construction. The use of standard components (FLAN-T5, Stable-Audio VAE) aids reproducibility. However, the code repository is not explicitly linked in the provided text (only a demo URL is given), and the exact preprocessing steps for the large-scale training data (5,650 hours) are summarized but not fully detailed in the snippet. The inclusion of ablation studies and specific metric values enhances the ability to verify claims.
A significant limitation is the reliance on generative modeling, which can introduce artifacts or "hallucinations" not present in the target source, although the CLAP$_A$ score suggests good fidelity. The model's performance on music separation (MUSDB18) is inferior to specialized models like SAM-Audio, likely due to the lack of music-specific training data. The computational cost of training and inference for DiT-based flow models is substantial, which may limit accessibility. Additionally, the "Self-Flow" adaptation requires careful tuning of the teacher-student update rates and flow time sampling strategies.
This work advances the field of audio source separation by demonstrating the efficacy of generative flow models for language-guided tasks. It offers a more flexible interface for audio editing and retrieval, allowing users to specify targets via natural language. The improved separation capabilities have applications in multimedia content creation, assistive listening technologies, and audio forensics. The open-source nature of the evaluation (via demo) and the use of public benchmarks contribute to the community's ability to benchmark future systems. FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
Contextualization is essential for production automatic speech recognition (ASR) systems, where user-provided phrases must be recognized accurately under strict latency constraints. Although many context-biasing methods improve recognition accuracy, they often do not address the practical requirements of modern production ASR systems: streaming inference, efficient batched decoding, user-specific context lists, and low runtime overhead. We propose TurboBias 2.0, a production-oriented framework for efficient phrase boosting in Transducer-based ASR systems. The framework extends GPU-accelerated TurboBias with a case-insensitive boosting graph and per-stream batched decoding, allowing each utterance in a batch to use an independent context-biasing configuration. This enables personalized context biasing for multiple simultaneous users without sharing or mixing their context lists. The proposed framework supports both offline and streaming inference and can be used with greedy and beam-search decoding. Experiments show that TurboBias 2.0 improves contextual phrase recognition while preserving low latency and high throughput.
Primary: NVIDIA
All Institutions: NVIDIA
The paper presents TurboBias 2.0, a production-oriented framework for efficient, case-insensitive, and per-stream context biasing in streaming Transducer-based ASR systems, demonstrating significant improvements in contextual phrase recognition accuracy while maintaining low latency and high throughput in batched inference scenarios.
The paper proposes TurboBias 2.0, an extension of a previous GPU-accelerated context-biasing framework for Transducer-based ASR systems. The technical contributions are primarily engineering optimizations rather than novel algorithmic discoveries. Specifically, it introduces a case-insensitive boosting graph using variative BPE representations to handle capitalization robustly without expanding the vocabulary size significantly. It also implements per-stream context biasing in batched decoding, allowing independent context lists for different streams within a single GPU batch, which is critical for multi-tenant production services. Finally, it extends offline beam search to streaming beam search for Transducers. The methodology is sound and addresses specific pain points in production ASR (latency, batching, casing), but the core concept of shallow fusion context biasing is well-established. The "novelty" lies in the efficient implementation details (tensor layout, caching strategies) rather than new theoretical insights.
The evaluation is conducted on the Contextual Earnings-22 benchmark and an internal medical dataset. The experiments compare TurboBias 2.0 against a CTC-Word Spotter (CTC-WS) baseline and no-context baselines. Results show significant improvements in Keyword F-score and WER, particularly when using beam search and per-stream context lists. The paper provides detailed ablation studies on case-insensitivity and per-stream vs. global context. The experimental setup is rigorous, using standard metrics (WER, RTFx) and realistic production-like constraints (streaming latency, batch size). However, the reliance on an internal medical dataset limits the generalizability of those specific results, although the Earnings-22 results are comparable to public benchmarks.
The paper provides detailed descriptions of the graph construction, tensor layouts, and decoding logic. It explicitly links to NVIDIA NeMo pull requests, suggesting code availability. The use of standard models (Parakeet, Nemotron) and public benchmarks (Earnings-22) aids reproducibility. The internal medical dataset is not public, which is a minor limitation for full reproducibility of that specific part of the evaluation.
The paper does not extensively discuss the memory overhead of maintaining hundreds of per-stream boosting graphs on GPU, although it claims it is manageable. The performance gain of beam search over greedy decoding is noted, but the computational cost increase is implicit rather than explicitly quantified in terms of additional latency vs. accuracy trade-off in a granular way. The method is specific to Transducer architectures and may not generalize directly to other ASR paradigms without adaptation.
This work has significant practical impact for the deployment of ASR systems in production environments, particularly for voice assistants, transcription services, and meeting analysis tools where low latency, high throughput, and personalized context are required. By enabling efficient per-stream biasing, it allows for more accurate and robust speech recognition in multi-user scenarios. The paper presents TurboBias 2.0, a production-oriented framework for efficient, case-insensitive, and per-stream context biasing in streaming Transducer-based ASR systems, demonstrating significant improvements in contextual phrase recognition accuracy while maintaining low latency and high throughput in batched inference scenarios.
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.
This paper provides a critical diagnostic analysis of benchmark contamination in Automatic Speech Recognition models, introducing novel behavioral probes to quantify how models prioritize reference transcripts over acoustic evidence. The work is highly relevant to the current landscape of LLM-audio integration and evaluation, as it addresses a fundamental validity issue in model assessment. The technical contribution is strong, offering mechanistic insights and causal manipulation techniques that advance the field's understanding of model generalization versus benchmark overfitting. While not a new generative architecture or a primary quality metric, it serves as a high-value evaluation study that will likely influence future benchmark design and model training practices.
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Primary: Imperial College London
All Institutions: Imperial College London, University of Surrey
[One sentence main contribution]. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
The paper proposes NAPE, a self-supervised learning framework for audio that adapts the "next-token prediction" paradigm from Large Language Models (LLMs) to continuous audio patch embeddings. The core innovation is a minimalist causal Transformer that predicts the next patch embedding in a log-mel spectrogram sequence using only causal masking and a stop-gradient operator on the target, avoiding reconstruction decoders, tokenizers, or student-teacher setups. The methodology is theoretically sound and aligns with recent trends in joint-embedding predictive architectures (like JEPA) but applies them to the temporal domain of audio. The design choices regarding scanning orders (Raster, Time-major, Zigzag, Diagonal) for the 2D spectrogram grid are a significant technical contribution, as they address the inductive bias required for causal prediction in non-isotropic data. The use of negative cosine similarity with stop-gradient is a known technique (from SimSiam/JSSE), so the novelty lies primarily in the application and specific architectural adaptations (scanning orders, causal masking) rather than a fundamentally new loss function.
The experimental evaluation is comprehensive, covering six standard audio and speech benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP). The authors demonstrate state-of-the-art or competitive performance on several tasks, particularly in fine-tuning scenarios. They also provide linear-probing results, which is crucial for assessing the quality of the learned representations independent of the downstream task head. The scaling analysis (Small, Base, Large) shows consistent performance improvements, suggesting the method is scalable. The ablation studies on scanning orders and predictor heads provide valuable insights into the design space. However, the paper lacks direct comparison with other autoregressive audio models (if any exist in the immediate literature) or joint-embedding methods specifically tailored for audio (like WavJEP), which would strengthen the claim of being a "markedly different" and superior philosophy. The results are strong, but the "state-of-the-art" claim should be contextualized against the specific baselines provided.
The paper provides a GitHub repository link and details the architectural components (Conv2d patch embedding, RoPE, LayerScale, etc.). The training objective is clearly defined with pseudocode. The pre-training data (AudioSet) and preprocessing steps (16kHz, 10s clips, 128 mel bands) are standard and well-documented. The inclusion of ablation studies on key hyperparameters (scanning order, predictor head) enhances reproducibility. The code availability is explicitly stated.
The paper does not explicitly discuss the computational cost of the causal masking compared to masked modeling or contrastive learning. Causal Transformers can be more memory-intensive during training due to the attention mask. The paper also does not explore the robustness of the scanning order across different audio domains (e.g., music vs. speech vs. environmental sounds) in depth, although it mentions it. The reliance on AudioSet for pre-training might limit generalization to out-of-distribution domains not well-represented in AudioSet.
This work contributes to the democratization of audio representation learning by providing a simpler, more unified framework that aligns with the dominant paradigm in NLP and Vision. This could lead to more efficient pre-training pipelines and better transfer learning capabilities for audio tasks. It also opens up new avenues for research into autoregressive audio generation and understanding by establishing a strong baseline for next-embedding prediction. [One sentence main contribution]. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.
Primary: ETH Zurich
All Institutions: ETH Zurich
The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
The paper proposes "Geometric Iterative Retrieval" (GIR), a novel framework for neural audio codec resynthesis. It correctly identifies a gap in the design space between discrete token prediction and continuous regression, positioning GIR in the "continuous, iterative" cell, distinct from diffusion methods by aligning its iterative steps with the RVQ layer hierarchy rather than a noise schedule. The methodology is technically sound: it uses a DeBERTa-v3 backbone to predict continuous latent vectors for each RVQ layer, supervised by a CLIP-style contrastive loss. A key innovation is the replacement of the standard additive RVQ decoding with a self-attention aggregator, allowing for non-additive combination of layer embeddings. This approach effectively leverages the geometric structure of the codebook space, addressing the "geometry-blind" nature of cross-entropy loss in token prediction and the "mean-seeking" behavior of MSE regression. The theoretical framing of the design space is clear and compelling.
The experimental setup is rigorous, evaluating on three diverse datasets (MTG-Jamendo, Common Voice, FMA) using the DAC codec. The baselines are comprehensive, including naive decoding, one-step regression (MSE and Cosine), discrete token prediction (CE), and ablations of the proposed method. The results show that GIR achieves the best Log-Spectral Distance (LSD) and is preferred in double-blind human listening tests over all baselines, including the strong one-step regression baseline. The layer-progression analysis reveals an interesting tension: objective metrics peak at K=3 layers, while human listeners prefer the full K=9 decode, suggesting current objective metrics may not fully capture perceptual quality for residual layers. This finding is valuable for the community. The ablation studies effectively isolate the contributions of the per-layer target, contrastive loss, and attention aggregator.
The paper provides sufficient implementation details for reproduction, including the backbone architecture (DeBERTa-v3), training hyperparameters (AdamW, learning rate schedule, batch size), and dataset mixtures. The code repository is linked. The use of standard codecs (DAC) and metrics (LSD, SI-SDR, FAD) further aids reproducibility.
The method is currently evaluated only on the DAC codec; generalization to other RVQ-based codecs (like EnCodec or SoundStream) is not demonstrated, though the authors claim it is codec-agnostic. The layer-progression analysis shows that adding more layers beyond K=3 degrades objective metrics, which contradicts human preference, indicating a potential misalignment in evaluation metrics or a limitation in how the model handles deep residual layers. The computational cost of the iterative inference (D-1 steps) is higher than one-step regression, though comparable to discrete token prediction.
This work significantly advances the field of neural audio generation by providing a high-fidelity resynthesis method that bridges the gap between discrete and continuous representations. High-quality resynthesis is critical for the fidelity of end-to-end audio generation systems. The insight that codec hierarchy can guide iterative refinement offers a new paradigm for designing generative models in other domains with hierarchical latent structures. The finding regarding the disconnect between spectral metrics and human preference has broader implications for audio evaluation benchmarking. The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institution, Yinwang Intelligent Technology Co., Ltd
VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
The paper proposes VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. The methodology addresses the misalignment of existing metric-based rewards by introducing a human-preference dataset (VAPref-10K) and a three-stage training pipeline: easy cold-start SFT, hard preference alignment via rejection sampling, and dimension-wise reinforcement learning (GRPO). The approach is technically sound, leveraging recent advances in LLM-based reward modeling (CoT, GRPO) and adapting them to the multimodal video-audio domain. The introduction of dimension-wise rewards to provide denser signals than binary labels is a logical and valuable contribution to the field of preference optimization.
The experimental evaluation is comprehensive. The authors construct a new benchmark (VA-Judger-Bench) with in-domain and out-of-domain splits. They demonstrate that VA-Judger outperforms standard metrics (VideoAlign, AudioBox, CLIP, etc.) in predicting human preferences. Crucially, they show that using VA-Judger for post-training (RLHF-style) the LTX-2 model leads to significant improvements in both automatic metrics (JavisBench) and human preference scores (62.3% win rate). The ablation studies on the training stages are well-presented.
The paper provides detailed implementation details, including hyperparameters, model architectures (Qwen3-Omni backbone), and training configurations (DeepSpeed ZeRO 3, BF16). The code repository is linked. The dataset construction process is described, though the specific human annotation protocol details are somewhat condensed. Overall, reproducibility is high.
The reliance on Gemini 3.1 Pro for generating initial CoT responses and the rejection sampling step introduces a dependency on a specific, powerful proprietary model, which may limit the accessibility of the data generation pipeline for some researchers. Additionally, the dataset size (10.3K pairs) is relatively small compared to large-scale web datasets, though sufficient for preference modeling. The generalization to closed-source models (Sora, Veo) is promising but relies on the availability of their outputs for evaluation.
This work significantly advances the field of multimodal generation by providing a robust, human-aligned reward signal for joint video-audio generation. It addresses the critical "reward hacking" problem inherent in metric-based optimization. The dataset and benchmark will serve as valuable resources for future research in multimodal alignment and evaluation. VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.