Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.
Primary: Xiaohongshu
All Institutions: Xiaohongshu
[This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
The paper proposes FireRedTTS3, a framework for continuous autoregressive text-to-speech (TTS) that addresses error accumulation through semantically enriched representations. The core innovation is the RedAE tokenizer, which uses a frozen, pretrained Audio Encoder as a semantic teacher to regularize the latent space during training. This approach aims to improve text-speech alignment and stability without adding complex architectural components or multi-stage training pipelines. The generation backbone utilizes an LLM-DiT (Diffusion Transformer) architecture, leveraging the instruction-following capabilities of large language models (Qwen3) for both zero-shot cloning (Base variant) and instruction-controlled editing/design (Instruct variant). The methodology is sound and addresses a known pain point in continuous autoregressive TTS (error accumulation), offering a relatively simple solution compared to prior works that require additional semantic modules or complex tokenizers.
The authors evaluate FireRedTTS3 on four benchmarks: Seed-TTS-Eval, MiniMax-MLS-Test, InstructTTSEval, and Ming-Freeform-Audio-Edit. The results claim state-of-the-art performance in terms of speech intelligibility, speaker similarity, and instruction-following capabilities. The evaluation covers multilingual zero-shot cloning, instruction-controlled voice design, and speech editing. The use of established benchmarks adds credibility. However, the reliance on automated metrics (WER/CER, WavLM similarity, LLM-as-a-judge) for instruction following introduces potential bias, although this is standard in the field. The claim of "best average" performance is supported by the provided tables (referenced as [REF]), suggesting strong empirical results.
The paper provides a GitHub link for code and models. The methodology describes the architecture (Qwen3-style Transformers, DiT, RedAE) and training objectives (GAN, flow-matching, semantic distillation). The use of pretrained components (Qwen3, CAM++, FireRedAudio) aids reproducibility. However, the specific details of the "FireRedAudio" model and the exact composition of the 500k hours of training data are not fully detailed, which may hinder exact replication. The description of the Aggregator and DiT modules is sufficiently detailed for implementation by researchers familiar with these architectures.
The paper acknowledges that Cantonese error rates are high due to the recognition model (Whisper-large-v3) limitations, not necessarily synthesis quality. The reliance on a frozen semantic teacher requires the existence of a high-quality pretrained audio understanding model, which may not be available for all languages or domains. The "simple" architecture still relies on large pretrained LLMs and DiTs, implying significant computational resources for training and inference. The generalization to unseen languages (Portuguese, Ukrainian) is claimed but not deeply analyzed regarding the source of this generalization (linguistic universality vs. acoustic similarity).
FireRedTTS3 contributes to the advancement of controllable and editable speech synthesis, which has applications in accessibility, entertainment, and content creation. The ability to perform instruction-controlled editing and voice design enhances user interaction with speech technologies. However, the ease of voice cloning and editing raises concerns about deepfakes and misuse for fraudulent activities. The open-source release of code and models necessitates careful consideration of safety guardrails and usage policies. [This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.
Primary: ETH Zurich
All Institutions: ETH Zurich
The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
The paper proposes "Geometric Iterative Retrieval" (GIR), a novel framework for neural audio codec resynthesis. It correctly identifies a gap in the design space between discrete token prediction and continuous regression, positioning GIR in the "continuous, iterative" cell, distinct from diffusion methods by aligning its iterative steps with the RVQ layer hierarchy rather than a noise schedule. The methodology is technically sound: it uses a DeBERTa-v3 backbone to predict continuous latent vectors for each RVQ layer, supervised by a CLIP-style contrastive loss. A key innovation is the replacement of the standard additive RVQ decoding with a self-attention aggregator, allowing for non-additive combination of layer embeddings. This approach effectively leverages the geometric structure of the codebook space, addressing the "geometry-blind" nature of cross-entropy loss in token prediction and the "mean-seeking" behavior of MSE regression. The theoretical framing of the design space is clear and compelling.
The experimental setup is rigorous, evaluating on three diverse datasets (MTG-Jamendo, Common Voice, FMA) using the DAC codec. The baselines are comprehensive, including naive decoding, one-step regression (MSE and Cosine), discrete token prediction (CE), and ablations of the proposed method. The results show that GIR achieves the best Log-Spectral Distance (LSD) and is preferred in double-blind human listening tests over all baselines, including the strong one-step regression baseline. The layer-progression analysis reveals an interesting tension: objective metrics peak at K=3 layers, while human listeners prefer the full K=9 decode, suggesting current objective metrics may not fully capture perceptual quality for residual layers. This finding is valuable for the community. The ablation studies effectively isolate the contributions of the per-layer target, contrastive loss, and attention aggregator.
The paper provides sufficient implementation details for reproduction, including the backbone architecture (DeBERTa-v3), training hyperparameters (AdamW, learning rate schedule, batch size), and dataset mixtures. The code repository is linked. The use of standard codecs (DAC) and metrics (LSD, SI-SDR, FAD) further aids reproducibility.
The method is currently evaluated only on the DAC codec; generalization to other RVQ-based codecs (like EnCodec or SoundStream) is not demonstrated, though the authors claim it is codec-agnostic. The layer-progression analysis shows that adding more layers beyond K=3 degrades objective metrics, which contradicts human preference, indicating a potential misalignment in evaluation metrics or a limitation in how the model handles deep residual layers. The computational cost of the iterative inference (D-1 steps) is higher than one-step regression, though comparable to discrete token prediction.
This work significantly advances the field of neural audio generation by providing a high-fidelity resynthesis method that bridges the gap between discrete and continuous representations. High-quality resynthesis is critical for the fidelity of end-to-end audio generation systems. The insight that codec hierarchy can guide iterative refinement offers a new paradigm for designing generative models in other domains with hierarchical latent structures. The finding regarding the disconnect between spectral metrics and human preference has broader implications for audio evaluation benchmarking. The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.
Primary: University of Surrey
All Institutions: University of Surrey, King's College London, Meta Superintelligence Labs
FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
The paper proposes FlowSep2, a language-queried audio source separation (LASS) system based on Rectified Flow Matching (RFM) with a Diffusion Transformer (DiT) backbone. The core methodological contribution is the integration of "Self-Flow," a self-supervised representation alignment technique adapted from Self-Flow (SRFM), into the generative separation framework. This involves a teacher-student EMA setup where the student predicts a noisy view with mixed flow times, while the teacher provides a cleaner, uniform view, enforcing semantic consistency in the latent space. The approach replaces traditional mask-based discriminative models with a generative paradigm that synthesizes the target source directly from noise, conditioned on mixture audio and text. The use of RFM allows for efficient sampling compared to standard DDPMs. The methodology is technically sound and builds logically on recent advances in flow matching and self-supervised representation learning.
The authors evaluate FlowSep2 on six benchmarks: AudioCaps, VGGSound, ESC-50, MUSDB18, and DCASE 2024 Task 9 (Synth and Real). They employ a comprehensive suite of metrics including FAD, CLAP Score, CLAP$_A$ Score, AudioBox Aesthetics, and SAJ (SAM Audio Judge), supplemented by human subjective evaluations. The results demonstrate state-of-the-art performance on most general LASS benchmarks (AudioCaps, VGGSound, DCASE), significantly outperforming previous discriminative and generative baselines like LASS-Net, AudioSep, and the original FlowSep. Notably, on MUSDB18, FlowSep2 performs competitively in a zero-shot setting despite being trained only on general sound data, whereas SAM-Audio (trained on proprietary music data) leads. The ablation studies effectively isolate the contributions of the DiT backbone, RFM vs. DDPM, and the Self-Flow alignment strategy, showing consistent improvements.
The paper provides detailed descriptions of the model architecture, training hyperparameters (learning rate, batch size, optimizer), and dataset construction. The use of standard components (FLAN-T5, Stable-Audio VAE) aids reproducibility. However, the code repository is not explicitly linked in the provided text (only a demo URL is given), and the exact preprocessing steps for the large-scale training data (5,650 hours) are summarized but not fully detailed in the snippet. The inclusion of ablation studies and specific metric values enhances the ability to verify claims.
A significant limitation is the reliance on generative modeling, which can introduce artifacts or "hallucinations" not present in the target source, although the CLAP$_A$ score suggests good fidelity. The model's performance on music separation (MUSDB18) is inferior to specialized models like SAM-Audio, likely due to the lack of music-specific training data. The computational cost of training and inference for DiT-based flow models is substantial, which may limit accessibility. Additionally, the "Self-Flow" adaptation requires careful tuning of the teacher-student update rates and flow time sampling strategies.
This work advances the field of audio source separation by demonstrating the efficacy of generative flow models for language-guided tasks. It offers a more flexible interface for audio editing and retrieval, allowing users to specify targets via natural language. The improved separation capabilities have applications in multimedia content creation, assistive listening technologies, and audio forensics. The open-source nature of the evaluation (via demo) and the use of public benchmarks contribute to the community's ability to benchmark future systems. FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues over multiple audio inputs, requiring models to identify degradation types, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and explain low-level acoustic phenomena in natural language. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. MRMAD reveals an important yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
Primary: Northeastern University
All Institutions: Northeastern University, Bose Corporation, Stony Brook University
[One sentence main contribution]. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
The paper introduces MRMAD, a benchmark designed to evaluate Large Audio-Language Models (LALMs) on their ability to perceive and reason about audio degradation. The methodology involves constructing a dataset of 8,400 multiple-choice questions across three domains (speech, music, sound) and nine degradation types (e.g., noise, reverberation, compression). The evaluation is framed as multi-turn dialogues where models must identify degradation types, compare severity between two clips, or rank three clips by severity. The approach is methodologically sound for its purpose: it isolates low-level perceptual capabilities from high-level semantic understanding by using clean references and controlled degradations. The multi-turn design is a clever solution to the problem of comparing multiple audio inputs without temporal localization confusion. However, the novelty is moderate as it is essentially a specialized benchmark addition to the growing list of audio evaluation suites, rather than proposing a new model architecture or fundamental algorithm.
The experimental setup is comprehensive, evaluating 18 representative LALMs, including open-source models (SALMONN, Qwen2-Audio, Audio Flamingo) and closed-source/reasoning models (GPT-4o, Gemini, Qwen3-Omni). The results clearly demonstrate that current state-of-the-art models perform near random chance on degradation type identification and struggle significantly with severity comparison and ranking. This provides a valuable diagnostic baseline. The evaluation covers a wide range of model sizes and types, strengthening the generalizability of the findings. The use of balanced sampling and specific distractor generation for multiple-choice questions adds rigor to the evaluation protocol.
The paper provides a clear data construction pipeline, specifying source datasets (VCTK, LibriSpeech, MUSDB18, etc.), degradation parameters, and question generation templates. The code and dataset are made available via a GitHub link, which significantly enhances reproducibility. The multi-turn format and template-based prompt generation are described in sufficient detail for others to replicate the evaluation.
A key limitation is the reliance on synthetic degradations applied to high-quality source audio. While this allows for controlled severity levels, it may not fully capture the complexity of real-world audio degradation, which often involves complex, non-linear interactions between noise sources and signal characteristics. Additionally, the multiple-choice format, while efficient for large-scale evaluation, may not fully capture the nuanced reasoning capabilities of models compared to open-ended generation tasks. The exclusion of certain degradations for specific domains (e.g., echo in music) is justified but limits the universality of the benchmark. Finally, the evaluation is limited to 18 models; as the field evolves rapidly, the benchmark will need continuous updating.
This work has significant implications for the development of robust audio-language models. By highlighting the gap in low-level audio perception, it directs future research towards improving model robustness to real-world acoustic conditions. This is crucial for applications in accessibility (e.g., audio description for visually impaired users in noisy environments), audio restoration, and high-fidelity communication systems. The benchmark serves as a necessary tool for the community to track progress in this underexplored area. [One sentence main contribution]. [The paper presents MRMAD, a novel multi-turn, multi-audio benchmark that systematically evaluates the ability of Large Audio-Language Models to perceive, compare, and reason about audio degradations across speech, music, and sound domains, revealing significant limitations in current models' low-level acoustic understanding].
Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.
Primary: University of Surrey
All Institutions: University of Surrey, King's College London, Meta Superintelligence Labs
FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
The paper proposes FlowSep2, a language-queried audio source separation (LASS) system based on Rectified Flow Matching (RFM) with a Diffusion Transformer (DiT) backbone. The core methodological contribution is the integration of "Self-Flow," a self-supervised representation alignment technique adapted from Self-Flow (SRFM), into the generative separation framework. This involves a teacher-student EMA setup where the student predicts a noisy view with mixed flow times, while the teacher provides a cleaner, uniform view, enforcing semantic consistency in the latent space. The approach replaces traditional mask-based discriminative models with a generative paradigm that synthesizes the target source directly from noise, conditioned on mixture audio and text. The use of RFM allows for efficient sampling compared to standard DDPMs. The methodology is technically sound and builds logically on recent advances in flow matching and self-supervised representation learning.
The authors evaluate FlowSep2 on six benchmarks: AudioCaps, VGGSound, ESC-50, MUSDB18, and DCASE 2024 Task 9 (Synth and Real). They employ a comprehensive suite of metrics including FAD, CLAP Score, CLAP$_A$ Score, AudioBox Aesthetics, and SAJ (SAM Audio Judge), supplemented by human subjective evaluations. The results demonstrate state-of-the-art performance on most general LASS benchmarks (AudioCaps, VGGSound, DCASE), significantly outperforming previous discriminative and generative baselines like LASS-Net, AudioSep, and the original FlowSep. Notably, on MUSDB18, FlowSep2 performs competitively in a zero-shot setting despite being trained only on general sound data, whereas SAM-Audio (trained on proprietary music data) leads. The ablation studies effectively isolate the contributions of the DiT backbone, RFM vs. DDPM, and the Self-Flow alignment strategy, showing consistent improvements.
The paper provides detailed descriptions of the model architecture, training hyperparameters (learning rate, batch size, optimizer), and dataset construction. The use of standard components (FLAN-T5, Stable-Audio VAE) aids reproducibility. However, the code repository is not explicitly linked in the provided text (only a demo URL is given), and the exact preprocessing steps for the large-scale training data (5,650 hours) are summarized but not fully detailed in the snippet. The inclusion of ablation studies and specific metric values enhances the ability to verify claims.
A significant limitation is the reliance on generative modeling, which can introduce artifacts or "hallucinations" not present in the target source, although the CLAP$_A$ score suggests good fidelity. The model's performance on music separation (MUSDB18) is inferior to specialized models like SAM-Audio, likely due to the lack of music-specific training data. The computational cost of training and inference for DiT-based flow models is substantial, which may limit accessibility. Additionally, the "Self-Flow" adaptation requires careful tuning of the teacher-student update rates and flow time sampling strategies.
This work advances the field of audio source separation by demonstrating the efficacy of generative flow models for language-guided tasks. It offers a more flexible interface for audio editing and retrieval, allowing users to specify targets via natural language. The improved separation capabilities have applications in multimedia content creation, assistive listening technologies, and audio forensics. The open-source nature of the evaluation (via demo) and the use of public benchmarks contribute to the community's ability to benchmark future systems. FlowSep2 establishes a strong generative baseline for language-queried audio source separation by effectively combining Rectified Flow Matching with self-supervised semantic representation alignment, achieving state-of-the-art results on multiple benchmarks and demonstrating robust generalization to complex acoustic scenes.
Contextualization is essential for production automatic speech recognition (ASR) systems, where user-provided phrases must be recognized accurately under strict latency constraints. Although many context-biasing methods improve recognition accuracy, they often do not address the practical requirements of modern production ASR systems: streaming inference, efficient batched decoding, user-specific context lists, and low runtime overhead. We propose TurboBias 2.0, a production-oriented framework for efficient phrase boosting in Transducer-based ASR systems. The framework extends GPU-accelerated TurboBias with a case-insensitive boosting graph and per-stream batched decoding, allowing each utterance in a batch to use an independent context-biasing configuration. This enables personalized context biasing for multiple simultaneous users without sharing or mixing their context lists. The proposed framework supports both offline and streaming inference and can be used with greedy and beam-search decoding. Experiments show that TurboBias 2.0 improves contextual phrase recognition while preserving low latency and high throughput.
Primary: NVIDIA
All Institutions: NVIDIA
The paper presents TurboBias 2.0, a production-oriented framework for efficient, case-insensitive, and per-stream context biasing in streaming Transducer-based ASR systems, demonstrating significant improvements in contextual phrase recognition accuracy while maintaining low latency and high throughput in batched inference scenarios.
The paper proposes TurboBias 2.0, an extension of a previous GPU-accelerated context-biasing framework for Transducer-based ASR systems. The technical contributions are primarily engineering optimizations rather than novel algorithmic discoveries. Specifically, it introduces a case-insensitive boosting graph using variative BPE representations to handle capitalization robustly without expanding the vocabulary size significantly. It also implements per-stream context biasing in batched decoding, allowing independent context lists for different streams within a single GPU batch, which is critical for multi-tenant production services. Finally, it extends offline beam search to streaming beam search for Transducers. The methodology is sound and addresses specific pain points in production ASR (latency, batching, casing), but the core concept of shallow fusion context biasing is well-established. The "novelty" lies in the efficient implementation details (tensor layout, caching strategies) rather than new theoretical insights.
The evaluation is conducted on the Contextual Earnings-22 benchmark and an internal medical dataset. The experiments compare TurboBias 2.0 against a CTC-Word Spotter (CTC-WS) baseline and no-context baselines. Results show significant improvements in Keyword F-score and WER, particularly when using beam search and per-stream context lists. The paper provides detailed ablation studies on case-insensitivity and per-stream vs. global context. The experimental setup is rigorous, using standard metrics (WER, RTFx) and realistic production-like constraints (streaming latency, batch size). However, the reliance on an internal medical dataset limits the generalizability of those specific results, although the Earnings-22 results are comparable to public benchmarks.
The paper provides detailed descriptions of the graph construction, tensor layouts, and decoding logic. It explicitly links to NVIDIA NeMo pull requests, suggesting code availability. The use of standard models (Parakeet, Nemotron) and public benchmarks (Earnings-22) aids reproducibility. The internal medical dataset is not public, which is a minor limitation for full reproducibility of that specific part of the evaluation.
The paper does not extensively discuss the memory overhead of maintaining hundreds of per-stream boosting graphs on GPU, although it claims it is manageable. The performance gain of beam search over greedy decoding is noted, but the computational cost increase is implicit rather than explicitly quantified in terms of additional latency vs. accuracy trade-off in a granular way. The method is specific to Transducer architectures and may not generalize directly to other ASR paradigms without adaptation.
This work has significant practical impact for the deployment of ASR systems in production environments, particularly for voice assistants, transcription services, and meeting analysis tools where low latency, high throughput, and personalized context are required. By enabling efficient per-stream biasing, it allows for more accurate and robust speech recognition in multi-user scenarios. The paper presents TurboBias 2.0, a production-oriented framework for efficient, case-insensitive, and per-stream context biasing in streaming Transducer-based ASR systems, demonstrating significant improvements in contextual phrase recognition accuracy while maintaining low latency and high throughput in batched inference scenarios.
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.
This paper provides a critical diagnostic analysis of benchmark contamination in Automatic Speech Recognition models, introducing novel behavioral probes to quantify how models prioritize reference transcripts over acoustic evidence. The work is highly relevant to the current landscape of LLM-audio integration and evaluation, as it addresses a fundamental validity issue in model assessment. The technical contribution is strong, offering mechanistic insights and causal manipulation techniques that advance the field's understanding of model generalization versus benchmark overfitting. While not a new generative architecture or a primary quality metric, it serves as a high-value evaluation study that will likely influence future benchmark design and model training practices.
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Primary: Imperial College London
All Institutions: Imperial College London, University of Surrey
[One sentence main contribution]. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
The paper proposes NAPE, a self-supervised learning framework for audio that adapts the "next-token prediction" paradigm from Large Language Models (LLMs) to continuous audio patch embeddings. The core innovation is a minimalist causal Transformer that predicts the next patch embedding in a log-mel spectrogram sequence using only causal masking and a stop-gradient operator on the target, avoiding reconstruction decoders, tokenizers, or student-teacher setups. The methodology is theoretically sound and aligns with recent trends in joint-embedding predictive architectures (like JEPA) but applies them to the temporal domain of audio. The design choices regarding scanning orders (Raster, Time-major, Zigzag, Diagonal) for the 2D spectrogram grid are a significant technical contribution, as they address the inductive bias required for causal prediction in non-isotropic data. The use of negative cosine similarity with stop-gradient is a known technique (from SimSiam/JSSE), so the novelty lies primarily in the application and specific architectural adaptations (scanning orders, causal masking) rather than a fundamentally new loss function.
The experimental evaluation is comprehensive, covering six standard audio and speech benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP). The authors demonstrate state-of-the-art or competitive performance on several tasks, particularly in fine-tuning scenarios. They also provide linear-probing results, which is crucial for assessing the quality of the learned representations independent of the downstream task head. The scaling analysis (Small, Base, Large) shows consistent performance improvements, suggesting the method is scalable. The ablation studies on scanning orders and predictor heads provide valuable insights into the design space. However, the paper lacks direct comparison with other autoregressive audio models (if any exist in the immediate literature) or joint-embedding methods specifically tailored for audio (like WavJEP), which would strengthen the claim of being a "markedly different" and superior philosophy. The results are strong, but the "state-of-the-art" claim should be contextualized against the specific baselines provided.
The paper provides a GitHub repository link and details the architectural components (Conv2d patch embedding, RoPE, LayerScale, etc.). The training objective is clearly defined with pseudocode. The pre-training data (AudioSet) and preprocessing steps (16kHz, 10s clips, 128 mel bands) are standard and well-documented. The inclusion of ablation studies on key hyperparameters (scanning order, predictor head) enhances reproducibility. The code availability is explicitly stated.
The paper does not explicitly discuss the computational cost of the causal masking compared to masked modeling or contrastive learning. Causal Transformers can be more memory-intensive during training due to the attention mask. The paper also does not explore the robustness of the scanning order across different audio domains (e.g., music vs. speech vs. environmental sounds) in depth, although it mentions it. The reliance on AudioSet for pre-training might limit generalization to out-of-distribution domains not well-represented in AudioSet.
This work contributes to the democratization of audio representation learning by providing a simpler, more unified framework that aligns with the dominant paradigm in NLP and Vision. This could lead to more efficient pre-training pipelines and better transfer learning capabilities for audio tasks. It also opens up new avenues for research into autoregressive audio generation and understanding by establishing a strong baseline for next-embedding prediction. [One sentence main contribution]. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.
Primary: ETH Zurich
All Institutions: ETH Zurich
The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
The paper proposes "Geometric Iterative Retrieval" (GIR), a novel framework for neural audio codec resynthesis. It correctly identifies a gap in the design space between discrete token prediction and continuous regression, positioning GIR in the "continuous, iterative" cell, distinct from diffusion methods by aligning its iterative steps with the RVQ layer hierarchy rather than a noise schedule. The methodology is technically sound: it uses a DeBERTa-v3 backbone to predict continuous latent vectors for each RVQ layer, supervised by a CLIP-style contrastive loss. A key innovation is the replacement of the standard additive RVQ decoding with a self-attention aggregator, allowing for non-additive combination of layer embeddings. This approach effectively leverages the geometric structure of the codebook space, addressing the "geometry-blind" nature of cross-entropy loss in token prediction and the "mean-seeking" behavior of MSE regression. The theoretical framing of the design space is clear and compelling.
The experimental setup is rigorous, evaluating on three diverse datasets (MTG-Jamendo, Common Voice, FMA) using the DAC codec. The baselines are comprehensive, including naive decoding, one-step regression (MSE and Cosine), discrete token prediction (CE), and ablations of the proposed method. The results show that GIR achieves the best Log-Spectral Distance (LSD) and is preferred in double-blind human listening tests over all baselines, including the strong one-step regression baseline. The layer-progression analysis reveals an interesting tension: objective metrics peak at K=3 layers, while human listeners prefer the full K=9 decode, suggesting current objective metrics may not fully capture perceptual quality for residual layers. This finding is valuable for the community. The ablation studies effectively isolate the contributions of the per-layer target, contrastive loss, and attention aggregator.
The paper provides sufficient implementation details for reproduction, including the backbone architecture (DeBERTa-v3), training hyperparameters (AdamW, learning rate schedule, batch size), and dataset mixtures. The code repository is linked. The use of standard codecs (DAC) and metrics (LSD, SI-SDR, FAD) further aids reproducibility.
The method is currently evaluated only on the DAC codec; generalization to other RVQ-based codecs (like EnCodec or SoundStream) is not demonstrated, though the authors claim it is codec-agnostic. The layer-progression analysis shows that adding more layers beyond K=3 degrades objective metrics, which contradicts human preference, indicating a potential misalignment in evaluation metrics or a limitation in how the model handles deep residual layers. The computational cost of the iterative inference (D-1 steps) is higher than one-step regression, though comparable to discrete token prediction.
This work significantly advances the field of neural audio generation by providing a high-fidelity resynthesis method that bridges the gap between discrete and continuous representations. High-quality resynthesis is critical for the fidelity of end-to-end audio generation systems. The insight that codec hierarchy can guide iterative refinement offers a new paradigm for designing generative models in other domains with hierarchical latent structures. The finding regarding the disconnect between spectral metrics and human preference has broader implications for audio evaluation benchmarking. The paper introduces Geometric Iterative Retrieval, a novel continuous-iterative resynthesis method for neural audio codecs that outperforms discrete and one-step baselines in both objective metrics and human preference, while highlighting critical limitations in current spectral evaluation metrics.
Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institution, Yinwang Intelligent Technology Co., Ltd
VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
The paper proposes VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. The methodology addresses the misalignment of existing metric-based rewards by introducing a human-preference dataset (VAPref-10K) and a three-stage training pipeline: easy cold-start SFT, hard preference alignment via rejection sampling, and dimension-wise reinforcement learning (GRPO). The approach is technically sound, leveraging recent advances in LLM-based reward modeling (CoT, GRPO) and adapting them to the multimodal video-audio domain. The introduction of dimension-wise rewards to provide denser signals than binary labels is a logical and valuable contribution to the field of preference optimization.
The experimental evaluation is comprehensive. The authors construct a new benchmark (VA-Judger-Bench) with in-domain and out-of-domain splits. They demonstrate that VA-Judger outperforms standard metrics (VideoAlign, AudioBox, CLIP, etc.) in predicting human preferences. Crucially, they show that using VA-Judger for post-training (RLHF-style) the LTX-2 model leads to significant improvements in both automatic metrics (JavisBench) and human preference scores (62.3% win rate). The ablation studies on the training stages are well-presented.
The paper provides detailed implementation details, including hyperparameters, model architectures (Qwen3-Omni backbone), and training configurations (DeepSpeed ZeRO 3, BF16). The code repository is linked. The dataset construction process is described, though the specific human annotation protocol details are somewhat condensed. Overall, reproducibility is high.
The reliance on Gemini 3.1 Pro for generating initial CoT responses and the rejection sampling step introduces a dependency on a specific, powerful proprietary model, which may limit the accessibility of the data generation pipeline for some researchers. Additionally, the dataset size (10.3K pairs) is relatively small compared to large-scale web datasets, though sufficient for preference modeling. The generalization to closed-source models (Sora, Veo) is promising but relies on the availability of their outputs for evaluation.
This work significantly advances the field of multimodal generation by providing a robust, human-aligned reward signal for joint video-audio generation. It addresses the critical "reward hacking" problem inherent in metric-based optimization. The dataset and benchmark will serve as valuable resources for future research in multimodal alignment and evaluation. VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.
Primary: Xiaohongshu
All Institutions: Xiaohongshu
[This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
The paper proposes FireRedTTS3, a framework for continuous autoregressive text-to-speech (TTS) that addresses error accumulation through semantically enriched representations. The core innovation is the RedAE tokenizer, which uses a frozen, pretrained Audio Encoder as a semantic teacher to regularize the latent space during training. This approach aims to improve text-speech alignment and stability without adding complex architectural components or multi-stage training pipelines. The generation backbone utilizes an LLM-DiT (Diffusion Transformer) architecture, leveraging the instruction-following capabilities of large language models (Qwen3) for both zero-shot cloning (Base variant) and instruction-controlled editing/design (Instruct variant). The methodology is sound and addresses a known pain point in continuous autoregressive TTS (error accumulation), offering a relatively simple solution compared to prior works that require additional semantic modules or complex tokenizers.
The authors evaluate FireRedTTS3 on four benchmarks: Seed-TTS-Eval, MiniMax-MLS-Test, InstructTTSEval, and Ming-Freeform-Audio-Edit. The results claim state-of-the-art performance in terms of speech intelligibility, speaker similarity, and instruction-following capabilities. The evaluation covers multilingual zero-shot cloning, instruction-controlled voice design, and speech editing. The use of established benchmarks adds credibility. However, the reliance on automated metrics (WER/CER, WavLM similarity, LLM-as-a-judge) for instruction following introduces potential bias, although this is standard in the field. The claim of "best average" performance is supported by the provided tables (referenced as [REF]), suggesting strong empirical results.
The paper provides a GitHub link for code and models. The methodology describes the architecture (Qwen3-style Transformers, DiT, RedAE) and training objectives (GAN, flow-matching, semantic distillation). The use of pretrained components (Qwen3, CAM++, FireRedAudio) aids reproducibility. However, the specific details of the "FireRedAudio" model and the exact composition of the 500k hours of training data are not fully detailed, which may hinder exact replication. The description of the Aggregator and DiT modules is sufficiently detailed for implementation by researchers familiar with these architectures.
The paper acknowledges that Cantonese error rates are high due to the recognition model (Whisper-large-v3) limitations, not necessarily synthesis quality. The reliance on a frozen semantic teacher requires the existence of a high-quality pretrained audio understanding model, which may not be available for all languages or domains. The "simple" architecture still relies on large pretrained LLMs and DiTs, implying significant computational resources for training and inference. The generalization to unseen languages (Portuguese, Ukrainian) is claimed but not deeply analyzed regarding the source of this generalization (linguistic universality vs. acoustic similarity).
FireRedTTS3 contributes to the advancement of controllable and editable speech synthesis, which has applications in accessibility, entertainment, and content creation. The ability to perform instruction-controlled editing and voice design enhances user interaction with speech technologies. However, the ease of voice cloning and editing raises concerns about deepfakes and misuse for fraudulent activities. The open-source release of code and models necessitates careful consideration of safety guardrails and usage policies. [This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
Large Audio Language Models (LALMs) have made rapid progress on standardized benchmarks, yet their deployment in practical media workflows, curation, archival indexing, and content distribution remains largely unrealized. We identify automated audio chapterization, the task of segmenting continuous audio streams into thematically coherent chapters, as a demanding and commercially consequential setting that exposes this gap. Chapterization is challenging because boundaries are defined less by objective acoustic events than by subjective editorial judgment, requiring models to reason sequentially over long acoustic contexts and approximate creator-authored boundary decisions. We present AudioChaps, a post-training framework for aligning end-to-end LALMs for this task via Group Relative Policy Optimization (GRPO) guided by Chain-of-Thought (CoT) reasoning. To support training and evaluation, we curate three datasets: AudioChaps-Alignment, derived from creator-annotated chapter boundaries on YouTube; AudioChaps-CoT, which provides structured supervision for well-formatted, high-quality, and evidence-grounded boundary reasoning; and AudioChaps-Eval, a held-out benchmark for audio chapterization. Applying GRPO directly without a Supervised Fine-Tuning (SFT) cold start, AudioChaps-R1-Zero already improves average F1 by 33 points over the state-of-the-art LALM Audio-Flamingo-3-Think. The AudioChaps framework produces our final aligned LALM, AudioChaps-R1, which improves average F1 by 49 points. These results demonstrate that GRPO-trained LALMs can reliably transform unstructured auditory streams into navigable, structured media. Our code, models, and dataset resources will be released upon acceptance at https://github.com/ta012/AudioChaps.
Primary: University of Surrey
All Institutions: University of Surrey, Huawei Noah's Ark Lab
The paper presents a well-executed study on aligning LALMs for audio chapterization using a novel synthetic CoT dataset and GRPO, demonstrating significant performance gains and contributing valuable resources to the field.
The paper proposes AudioChaps, a framework for aligning Large Audio Language Models (LALMs) for automated audio chapterization. The core methodological contribution is a two-stage post-training pipeline: (1) Supervised Fine-Tuning (SFT) on a synthetic Chain-of-Thought (CoT) dataset (AudioChaps-CoT) to establish a structured reasoning format, and (2) Group Relative Policy Optimization (GRPO) to calibrate boundary decisions against creator-authored annotations. The construction of the CoT dataset via an "audio-to-text modality bridge" using a stronger model (Step-Audio-R1) to generate perception logs and reasoning traces is a notable engineering approach to address the lack of reasoning data for subjective audio tasks. The use of GRPO with rule-based rewards (format and accuracy) is a standard application of recent RL techniques but applied effectively to the audio domain. The approach is sound and addresses a specific gap in LALM capabilities (subjective segmentation vs. objective event detection).
The experimental setup is robust. The authors curate three datasets (Alignment, CoT, Eval) stratified across four acoustic regimes (structured speech, dynamic media, gaming, music). They evaluate on both clip-level binary classification and full-length recording chapter detection. The results show significant improvements over the base model (AF3-Think-8B) and a larger baseline (Step-Audio-R1-32B). The ablation studies effectively isolate the contributions of SFT and GRPO, demonstrating that SFT improves recall while GRPO improves precision. The comparison with an ASR-LLM cascade highlights the advantage of end-to-end audio processing for non-speech cues. The inclusion of human evaluation for reasoning quality adds value. The statistical significance testing via bootstrap is a good practice.
The paper provides detailed descriptions of the dataset construction, training prompts, and hyperparameters. The code and models are promised upon acceptance. The use of standard open-source models (AF3-Think, MOSS-Think) as backbones enhances reproducibility. The synthetic data generation pipeline is described clearly enough to be replicated.
The primary limitation is the reliance on synthetic CoT data, which may contain biases or errors from the teacher model. The task is formulated as a binary boundary detection within a window, which may not capture global context as effectively as native long-context models. The evaluation is limited to YouTube content, which may not generalize to all audio types (e.g., pure music without speech, highly noisy environments). The "creator-annotated" ground truth is a proxy for editorial judgment and may vary in quality or consistency.
This work contributes to the practical deployment of LALMs in media workflows, enabling better content navigation and archival indexing. It demonstrates the potential of RL alignment for subjective audio understanding tasks. The released datasets and benchmarks will facilitate further research in audio chapterization and reasoning. The paper presents a well-executed study on aligning LALMs for audio chapterization using a novel synthetic CoT dataset and GRPO, demonstrating significant performance gains and contributing valuable resources to the field.
Semantic speech tokens should preserve linguistic content while suppressing speaker- and duration-dependent variation inherited from acoustic inputs. We propose Iterative Semantic Token Purification (ISTP), an alternating speech-to-unit (S2U) and text-to-unit (T2U) training procedure guided by text predictability. Starting from an initial S2U tokenizer, each iteration trains a T2U model on its deduplicated token sequences. The decoded T2U predictions then serve as connectionist temporal classification targets for a newly initialized S2U model, whose outputs supervise the next T2U model. This cycle progressively aligns the two token generators and biases the token space toward information recoverable from text. Experiments on Mandarin and English show substantially improved S2U--T2U agreement. Independently trained de-tokenizers further show that the refined S2U and T2U tokens retain sufficient content for high-intelligibility voice conversion and text-to-speech synthesis. In voice conversion, the generated speaking rate follows the reference more closely. The refined tokens also exhibit substantially improved cross-speaker consistency and reduced probe-recoverable speaker information.
Primary: City University of Hong Kong
All Institutions: City University of Hong Kong, Huawei Leibniz Research Center, Chinese University of Hong Kong
[One sentence main contribution]. The paper introduces an iterative refinement procedure that aligns speech and text tokenizers to produce speaker-invariant semantic units. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This research presents a robust method for enhancing the semantic purity of speech tokens by leveraging the inductive bias of text-to-unit models. By iteratively filtering S2U outputs through a T2U model conditioned only on text, the method effectively strips away speaker and duration variations, resulting in tokens that are more consistent across speakers and languages. The comprehensive evaluation, including independent de-tokenizer testing and speaker probing, strongly validates the approach's effectiveness in balancing content preservation with speaker invariance. This contributes meaningfully to the field of speech tokenization by offering a practical solution to the long-standing challenge of acoustic variability in semantic representations.
The paper proposes Iterative Semantic Token Purification (ISTP), a method to align speech-to-unit (S2U) and text-to-unit (T2U) tokenizers. The core innovation is an alternating training loop: an initial S2U tokenizer generates tokens, which are deduplicated and used to train a T2U model. The T2U model then generates pseudo-targets from text, which are used to train a *newly initialized* S2U model via CTC. This cycle repeats, theoretically forcing the S2U tokenizer to discard speaker-specific and duration-dependent variations that are not recoverable from text. The methodology is logically sound and addresses a genuine problem in speech tokenization: the mismatch between acoustic variability and linguistic invariance. However, the approach is essentially a sophisticated form of self-distillation or knowledge transfer using a text-conditioned model as a "filter." While effective, the conceptual novelty is moderate as it relies on established components (HuBERT, FSQ, BART, CTC) in a novel iterative arrangement.
The experiments are comprehensive and well-designed. The authors evaluate on both Mandarin and English, using multiple metrics: S2U-T2U agreement (WER/BLEU), generation utility (TTS/VC WER/SIM), cross-speaker consistency (UED, SelfBLEU-4), and speaker privacy (probe accuracy). The results show significant improvements in consistency and speaker invariance without degrading content preservation. The use of independent de-tokenizers for evaluation is a strong methodological choice that isolates token quality from decoder bias. The inclusion of speaking rate analysis provides additional insight into duration normalization. The baselines are relevant (StableToken, R-Spin, FlexiCodec, etc.). The results are convincing and support the claims.
The paper provides sufficient detail regarding the architecture (HuBERT, FSQ, BART, CTC) and training data sources (LibriSpeech, GigaSpeech, AISHELL-2, etc.). The iterative procedure is clearly defined. However, specific hyperparameters for the BART model, learning rates, and the exact deduplication logic (beyond "consecutive repetitions") are not fully detailed in the text provided. The claim of "8,000 hours" of training data is substantial, but the reproducibility relies on the availability of the specific SOTA components (DSA-Tokenizer, CA-F5-TTS) and the precise implementation of the iterative loop.
The paper does not explicitly discuss the computational cost of the iterative process, which involves training multiple models from scratch in each iteration. This could be a significant barrier to adoption. Additionally, the method relies on the quality of the initial T2U model; if the initial S2U tokens are too noisy or misaligned, the T2U model might learn incorrect mappings. The paper assumes that "text-predictable" content is the only desirable content, which might discard useful prosodic or emotional cues that are not strictly phonetic but are recoverable from context in some T2U models.
This work has significant implications for privacy-preserving speech technologies and efficient speech generation. By reducing speaker information in tokens, it mitigates privacy risks. The improved consistency aids in building more robust multilingual speech systems. The potential for misuse in creating deepfakes with high linguistic fidelity but anonymized voices is a minor concern, but the primary impact is positive for privacy and efficiency. [One sentence main contribution]. The paper introduces an iterative refinement procedure that aligns speech and text tokenizers to produce speaker-invariant semantic units. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This research presents a robust method for enhancing the semantic purity of speech tokens by leveraging the inductive bias of text-to-unit models. By iteratively filtering S2U outputs through a T2U model conditioned only on text, the method effectively strips away speaker and duration variations, resulting in tokens that are more consistent across speakers and languages. The comprehensive evaluation, including independent de-tokenizer testing and speaker probing, strongly validates the approach's effectiveness in balancing content preservation with speaker invariance. This contributes meaningfully to the field of speech tokenization by offering a practical solution to the long-standing challenge of acoustic variability in semantic representations.
Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or decide when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap). The framework uses multi-turn interaction between a Composer and an Instruct model to form a closed evidence-acquisition loop. A Captioner first produces an initial description. Conditioned on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. For credit assignment in variable-length interactions, LOOP-GRPO replaces the trajectory-wide scalar advantage with span-aligned signals: leave-one-out contributions of individual questions to the accumulated evidence, a quality-cost utility for stopping, and an evidence-preservation utility for final synthesis. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the roles to co-evolve. ACE-Cap thus turns captioning from passive one-shot generation into an adaptive process that learns what evidence to acquire, when to stop, and how to preserve it in a long-paragraph caption.
Primary: Kling Team
All Institutions: Kling Team
ACE-Cap introduces a novel agentic framework for fine-grained audio captioning that leverages active evidence acquisition and a specialized reinforcement learning algorithm (LOOP-GRPO) to improve factual accuracy and detail recovery, representing a significant step forward in adaptive audio understanding systems.
The paper proposes ACE-Cap, a framework that reformulates long-paragraph fine-grained audio captioning as an active evidence acquisition problem. The core innovation lies in the agentic loop involving a text-only Composer and an audio-conditioned Instruct model. The Composer identifies missing acoustic details by querying the Instruct model, which provides grounded answers. This addresses the limitation of passive one-shot captioners. The training methodology introduces LOOP-GRPO (Leave-One-Out Per-turn GRPO), which assigns credit to individual questions based on their counterfactual contribution to the final evidence state, rather than using a trajectory-wide scalar advantage. This is a significant methodological contribution to reinforcement learning for agentic systems, specifically addressing the credit assignment problem in variable-length interaction trajectories. The use of a frozen caption-only judge and gold-grounded multiple-choice questions for reward modeling is a practical approach to aligning the agent with fine-grained factual accuracy.
The experimental evaluation is comprehensive, covering multiple benchmarks (Omni-Cloze, MMAR, MMAU, MMSU). The paper demonstrates that ACE-Cap outperforms strong open-source baselines and remains competitive with proprietary models like GPT-4o Audio and Gemini 2.5 Pro. The ablation studies effectively isolate the contributions of the different training stages (Captioner GRPO, Instruct GRPO, Composer LOOP-GRPO, and alternating co-evolution), showing monotonic improvement. The analysis of the Composer's query behavior and the impact of query budget provides valuable insights into the agent's decision-making process. The results support the claim that active evidence acquisition improves fine-grained captioning accuracy.
The paper provides detailed descriptions of the methodology, including the reward function, the LOOP-GRPO algorithm, and the training pipeline. The use of publicly available models (Qwen2.5-Omni-7B, Qwen3-8B) and datasets (ASID-1M) enhances reproducibility. However, the reliance on a proprietary model (Gemini 3.1 Pro) for generating gold captions and MCQs during the offline stage may introduce some variability, although the core training and evaluation are performed on open-source components. The code is not explicitly mentioned as open-sourced in the text, which is a minor drawback for immediate reproducibility.
The paper acknowledges limitations such as the restriction to audio-only inputs and the fixed caption-as-evaluation protocol. The reliance on a fixed question bank derived from gold captions means the reward signal cannot detect unsupported claims outside the predefined set. Additionally, the alternating optimization strategy, while stabilizing training, may not fully capture the complex interdependencies between the Composer and Instruct models compared to joint optimization, although the authors argue it prevents instability. The performance gap with the strongest proprietary models (e.g., Gemini 3.1 Pro on MMAR) suggests there is still room for improvement in handling very complex audio scenes.
This work contributes to the field of audio-language models by introducing an agentic paradigm for fine-grained understanding. The techniques developed for active evidence acquisition and credit assignment in multi-turn interactions have broader applications in other domains requiring iterative reasoning and information gathering, such as visual question answering or document QA. The emphasis on factual accuracy and evidence preservation aligns with the growing need for reliable and interpretable AI systems. ACE-Cap introduces a novel agentic framework for fine-grained audio captioning that leverages active evidence acquisition and a specialized reinforcement learning algorithm (LOOP-GRPO) to improve factual accuracy and detail recovery, representing a significant step forward in adaptive audio understanding systems.
Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmark for instruction-aware speech retrieval, in which natural-language instructions dynamically specify relevance criteria, including semantic content, speaker identity, speaking style, environmental sounds, and their combinations. We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models. Our results reveal that no current method robustly handles all retrieval intents. Text-based approaches perform relatively better at semantic retrieval but struggle with paralinguistic attributes, while speech-based models are moderately better at capturing acoustic properties but falter at following instructions. These findings highlight the need for unified architectures capable of instruction-aware speech retrieval.
Primary: National Taiwan University
All Institutions: National Taiwan University, NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
INSPIRE is the first benchmark for instruction-aware speech retrieval, systematically evaluating the ability of various models to follow natural-language instructions for retrieving speech based on semantic, speaker, style, and environmental attributes, revealing a significant gap in current methods' ability to handle multi-attribute, paralinguistic retrieval.
The paper proposes INSPIRE, a benchmark for instruction-aware speech retrieval. The methodology involves constructing four subsets (DailyTalk, VCTK, Expresso, Synthetic) to test retrieval across semantic, speaker, style, and environmental attributes. The core technical contribution is the formalization of the problem and the construction of the dataset, particularly the synthetic subset which uses GPT-4o-mini for TTS and GPT-5.2 for instruction generation. The evaluation methodology involves testing four distinct retrieval paradigms: Large Audio-Language Models (LALMs), Cascaded Pipelines (ASR/Captioning + Text Retrieval), Self-Supervised Speech Models, and Contrastive Audio-Language Models. The approach is sound in its comparative design, aiming to highlight the gaps in current models. However, the novelty is somewhat limited by the fact that it is primarily a benchmarking paper rather than proposing a new architecture or algorithm. The use of proprietary models (GPT-4o-mini, GPT-5.2) for data generation raises questions about the independence of the benchmark from specific model biases, although this is common in modern AI research.
The experimental evaluation is comprehensive in scope, covering a wide range of existing models and retrieval strategies. The results clearly demonstrate the "modality specialization gap": text-based methods excel at semantic retrieval but fail at paralinguistic tasks, while speech-based models do the opposite. The inclusion of ablation studies on instruction types, pooling strategies, and layer-wise analysis adds depth. The use of Recall@K and NDCG is standard and appropriate. The results are robustly presented with clear tables and radar plots. The finding that no current method handles all intents robustly is significant and well-supported by the data. The synthetic subset's quality assessment (WER, speaker accuracy, emotion accuracy) provides confidence in the dataset's validity.
The paper provides a GitHub link to the code and data, which is a strong point for reproducibility. The dataset construction process is described in detail, including the sources (DailyTalk, VCTK, Expresso) and the generation pipeline for the synthetic subset. The baseline implementations are standard and well-documented. The use of open-source models for most baselines (HuBERT, WavLM, CLAP, E5-Mistral, etc.) ensures that other researchers can reproduce the experiments. The reliance on proprietary models for some baselines (Gemini, GPT-4o-mini) limits full reproducibility of those specific comparisons but does not invalidate the overall benchmark.
A key limitation is the reliance on synthetic data for the most complex multi-attribute tasks. While the synthetic data is high-quality, it may not fully capture the nuances and complexities of real-world speech, such as overlapping speakers, extreme noise, or diverse accents. The instruction generation using GPT-5.2 (a hypothetical or very recent model name, possibly a typo for GPT-4o or similar) introduces a potential bias if the instructions are not diverse enough or if they reflect the specific linguistic patterns of the generator. Additionally, the benchmark is static; it does not address the dynamic nature of user queries in real-world applications. The performance of LALMs is still relatively low, suggesting that the field is in its early stages, which is a limitation of the current state-of-the-art rather than the paper itself, but it highlights the difficulty of the task.
This paper has significant broader impact by defining a new and challenging task in speech processing. It highlights the limitations of current multimodal models and provides a clear direction for future research towards unified, instruction-aware speech retrieval systems. This could lead to more intuitive and powerful speech search engines, accessibility tools, and archival systems. The benchmark will likely spur further research in this area, fostering competition and innovation. The emphasis on paralinguistic attributes (style, emotion, speaker) is particularly important for applications requiring nuanced understanding of human communication. INSPIRE is the first benchmark for instruction-aware speech retrieval, systematically evaluating the ability of various models to follow natural-language instructions for retrieving speech based on semantic, speaker, style, and environmental attributes, revealing a significant gap in current methods' ability to handle multi-attribute, paralinguistic retrieval.
Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is never observed during training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.
Primary: Kuaishou Technology
All Institutions: Kuaishou Technology
SingDance presents a novel and effective framework for compositional zero-shot singing-and-dancing video generation by introducing role-aware audio conditioning and asymmetric staged training, achieving strong performance in motion-beat alignment and lip synchronization while maintaining high visual fidelity.
The paper proposes SingDance, a video diffusion framework that addresses the compositional zero-shot problem of generating singing-and-dancing videos. The core methodological contribution is the formulation of vocal articulation as a semantic "role" (source vs. listener) rather than a simple binary lip-sync instruction. This allows the model to distinguish between a subject producing vocals and a subject listening to off-screen vocals. The architecture utilizes hard-compact routing to select relevant audio tokens (speech from Wav2Vec 2.0, music from MuQ) and injects them via frame-wise joint audio injection into a Video DiT backbone (Wan2.2). A key innovation is the staged training strategy with asymmetric supervision: role control is learned from speech data, and music-conditioned dance is learned from instrumental/dancing-only data, without ever seeing paired singing-and-dancing data during training. The zero-shot composition at inference (Song/Source) combines these separately learned capabilities. The approach is technically sound and elegantly solves the conditioning conflict between speech and music in a unified framework.
The experimental evaluation is comprehensive. The authors curate specific test sets (SingDance-50, Dance-100) to evaluate the proposed capabilities. They compare against strong baselines including MusicInfuser and Wan-S2V. The results demonstrate strong motion-beat alignment and visual fidelity. Crucially, the paired vocal-role switching experiment effectively verifies that the model can switch between singing and listening while maintaining consistent body motion and audio conditioning, a capability that baseline models lacking explicit role conditioning fail to achieve robustly. The lip synchronization metrics are competitive with specialized speech-driven models, despite the parameter efficiency. The ablation studies support the necessity of the music pathway for beat alignment.
The paper provides detailed implementation details, including the backbone (Wan2.2-TI2V-5B), training hardware (64 A100s), hyperparameters (learning rate, batch size), and specific audio encoders (Wav2Vec 2.0, MuQ). The data sources are described (HuMoSet, MA-Data, proprietary collections), though the proprietary nature of some data may limit full reproducibility of the training set. The project page URL is provided, which likely contains code or weights, enhancing reproducibility potential.
The current model is limited to short, single-person clips with a binary, clip-level vocal role. It does not handle within-clip role transitions, duets, or multiple vocal roles. The reliance on hard-compact routing means the model must be explicitly told the role; it does not automatically infer the role from the audio content, which is a design choice but also a limitation for fully autonomous generation. The use of proprietary data for training is a constraint for independent verification.
This work significantly advances the field of multimodal video generation, particularly in the niche of music and speech-conditioned human animation. By enabling compositional zero-shot singing-and-dancing, it lowers the barrier for creating high-quality, personalized performance videos. The role-aware conditioning framework could be extended to other interactive scenarios. However, the ability to generate realistic singing and dancing videos also raises concerns regarding deepfake misuse, although the explicit role conditioning might offer some avenue for detection or control. SingDance presents a novel and effective framework for compositional zero-shot singing-and-dancing video generation by introducing role-aware audio conditioning and asymmetric staged training, achieving strong performance in motion-beat alignment and lip synchronization while maintaining high visual fidelity.