This week’s shortlist
Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per…
Abstract
Generating long-form speech requires models to keep content, speaker identity, turn-taking, and expressive style consistent over time. Yet most open training corpora contain examples lasting only tens of seconds and open long-form benchmarks cover at most two speakers per recording. We introduce Emilia2, a 229K-hour multilingual dataset for controllable long-form speech generation. From in-the-wild data, Emilia2 constructs short utterances, single-speaker long-form speech, and multi-speaker dialogues, preserving transcripts, speaker identities, pauses, and acoustic transitions. Examples longer than five minutes account for 50.0% of single-speaker long-form hours and 66.8% of dialogue hours. To support expressive control, Emilia2 provides natural language instructions for 4.77M examples and nonverbal-vocalization tags for 933K examples. We also introduce Emilia2-testset, 23 hours of bilingual recordings of up to 30 minutes and four speakers, each speaker paired with reference audio. We train autoregressive (AR) and non-autoregressive (NAR) models on Emilia2. Our 0.6B-backbone NAR model achieves the lowest Chinese CER and English WER on TTSD-Eval (5.27%/9.93%) among compared systems, while our AR model is on par with Qwen3-TTS-12Hz-VD on InstructTTSEval (zh/en: 76.2/78.3 vs. 77.1/77.9). These results validate Emilia2 as a comprehensive data foundation for long-form speech generation and minute-scale expressive control. Demo: https://charlesnii.github.io/Emilia2-demo/.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces Emilia2, a 229K-hour multilingual dataset and benchmark for controllable long-form speech generation, demonstrating that models trained on this data achieve state-of-the-art performance in both long-form consistency and expressive control.
Methodology
The paper proposes a comprehensive data construction pipeline, Emilia2, that transforms in-the-wild audio into a structured dataset for long-form speech generation. The methodology is robust, utilizing a multi-stage process involving audio standardization, speaker diarization (DiariZen), VAD (FireRedVAD), and quality filtering (DNSMOS). A key methodological strength is the preservation of temporal continuity and acoustic transitions in long-form examples, rather than concatenating trimmed segments, which is crucial for modeling natural pauses and turn-taking. The integration of LLM-based annotation (Qwen3-Omni) for style instructions and specialized ASR for nonverbal vocalizations (NV) adds significant value for controllable generation. The training of both Autoregressive (AR) and Non-Autoregressive (NAR) models on this data validates the dataset's versatility.
Results & evaluation
The experimental evaluation is extensive, covering long-form generation (TTSD-Eval, Long-TTS-Eval, Emilia2-testset) and short-form controllable generation (InstructTTSEval, NV-Bench). The introduction of Emilia2-testset, featuring up to 30-minute recordings with up to 4 speakers and per-speaker reference audio, addresses a critical gap in existing benchmarks. The results show that the 0.6B backbone models trained on Emilia2 outperform larger baselines (e.g., 3B Higgs Audio v2) in objective metrics like WER/CER and speaker similarity. Subjective evaluations (MOS) further support the naturalness and control capabilities. The comparison against state-of-the-art systems like Qwen3-TTS and MOSS-TTSD is relevant and competitive.
Reproducibility
The paper provides detailed descriptions of the pipeline components (specific models used for diarization, ASR, etc.) and the training curriculum. However, the source data is described as "in-the-wild" from various internet sources, which may raise questions about licensing and exact reproducibility of the raw data collection. The release of the dataset and testset is a major positive for reproducibility. The code for the models is not explicitly linked in the provided text, but the demo page is available.
Limitations
The primary limitation is the reliance on in-the-wild data, which may contain inherent biases or quality issues despite filtering. The annotation of instructions and NV tags is limited to segments shorter than 5 minutes, which might restrict the granularity of control for very long, continuous expressive shifts. Additionally, the paper focuses heavily on Chinese and English for NV annotations, with other languages potentially having less expressive control support.
Broader impact
This dataset has significant potential to advance the field of long-form speech synthesis, enabling more realistic audiobooks, podcasts, and conversational agents. By providing a standardized benchmark with multi-speaker and long-duration support, it facilitates fair comparison of new architectures. The focus on controllable generation (style, NV) opens avenues for more expressive and human-like AI interactions. The paper introduces Emilia2, a 229K-hour multilingual dataset and benchmark for controllable long-form speech generation, demonstrating that models trained on this data achieve state-of-the-art performance in both long-form consistency and expressive control.
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature…
Abstract
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled $2\times2$ experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40\% of its trainable parameters.
AI-generated analysis. Check claims against the paper.
Summary
The paper demonstrates that a frozen general-audio codec can support spatial reasoning by addressing audio-text information asymmetry through explicit inter-channel features and comprehensive scene supervision, achieving superior dual-source localization performance with reduced parameter count.
Methodology
The paper proposes AURA, a framework for spatial audio understanding that leverages a frozen general-audio codec (X-Codec) rather than a dedicated spatial encoder. The methodology is grounded in a rigorous layer-wise analysis of the codec's representations, identifying that spatial cues (specifically lateral cues) are preserved in deeper RVQ layers and are best accessed via inter-channel differences. The core innovation is the "information asymmetry" hypothesis, which posits that standard QA training fails because the audio contains multiple sources while the text target describes only one. To address this, AURA introduces two components: (1) a representation-side intervention adding explicit inter-channel difference slots to the audio tokens, and (2) a text-side intervention using "scene supervision" where the model must describe all sources before answering. The approach is technically sound, effectively bridging the gap between general audio representations and specific spatial reasoning tasks without retraining the heavy backbone models.
Results & evaluation
The experiments are well-controlled, particularly the 2x2 ablation study that isolates the effects of scene supervision and inter-channel differences. The results on SpatialSoundQA show a significant improvement (10.31 points) over the strongest dedicated system in dual-source direction accuracy, which is a strong claim given the parameter efficiency (40% of trainable parameters). The evaluation correctly focuses on the challenging dual-source scenarios where existing models fail. However, the reliance on a single benchmark (SpatialSoundQA) and the lack of evaluation on other spatial datasets (like BiDepth or real-world recordings) slightly limits the generalizability of the findings.
Reproducibility
The paper provides detailed descriptions of the architecture modifications (6-slot representation, LoRA modules) and the training strategy. However, specific hyperparameters, learning rates, and exact implementation details for the "geometric supervision" prompt construction are not fully detailed in the provided text. The use of standard components (Qwen3-8B, X-Codec) aids reproducibility, but the specific prompt engineering for scene supervision is a critical component that requires careful replication.
Limitations
The primary limitation is the dependence on simulated binaural data (SpatialSoundQA) for training and evaluation. It is unclear how well the model generalizes to real-world binaural recordings with complex noise and reverberation. Additionally, the model is frozen, which means it cannot adapt to new spatial concepts or room types beyond what the codec has seen. The "scene supervision" approach increases the length of the model's output, which may have computational implications for real-time applications.
Broader impact
This work has significant implications for the development of efficient spatial audio models. By demonstrating that general audio codecs retain spatial information, it challenges the necessity of dedicated spatial encoders, potentially simplifying the architecture of future audio-language models. The insights into information asymmetry in multi-source scenes are applicable to other multimodal tasks where the input contains more information than the target output. This could influence how datasets are constructed and how models are trained for complex scene understanding. The paper demonstrates that a frozen general-audio codec can support spatial reasoning by addressing audio-text information asymmetry through explicit inter-channel features and comprehensive scene supervision, achieving superior dual-source localization performance with reduced parameter count.
We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio…
Abstract
We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding. Our main contribution is not a new architecture but a data-centric training recipe; we use a standard decoder-only backbone initialized from Gemma 4, so that gains can be traced to the data. For speech generation, Walrus predicts low-bitrate discrete speech tokens, and a lightweight token-to-waveform decoder converts them into high-fidelity audio. The recipe is a three-stage data curriculum that moves from broad multilingual data to multi-talker, code-switching, and domain-specific data. At each stage, task-aware and dataset-aware weighting balances tasks with very different token budgets and protects core skills from forgetting. Trained on over 600M samples covering more than 20 tasks and 100 languages, Walrus leads on most multilingual, multi-talker, and code-switching ASR, speech translation, and audio understanding benchmarks, outperforming both open speech language models of comparable size and dedicated task-specific models. On most benchmarks, it even matches or surpasses the $4\times$ larger Qwen3-Omni's performance. Results suggest that data engineering is a primary driver of quality in multi-task speech and audio modeling.
AI-generated analysis. Check claims against the paper.
Summary
The main contribution is a data-centric training recipe for a unified 8B speech-and-audio foundation model that achieves state-of-the-art results across diverse tasks without architectural novelty. By fixing the backbone and focusing on a three-stage data curriculum with task-aware weighting, the paper demonstrates that data engineering is a primary driver of quality in multi-task speech modeling, outperforming larger models and dedicated task-specific systems on key benchmarks for ASR, translation, and audio understanding.
Methodology
The paper proposes a data-centric approach to training a unified speech-and-audio foundation model, explicitly arguing that data engineering is more critical than architectural novelty. The methodology relies on a standard decoder-only Transformer backbone (Gemma 4 E4B) initialized with pre-trained weights, keeping the architecture fixed to isolate the impact of the data recipe. The core innovation lies in a three-stage data curriculum: Stage 1 establishes a broad multilingual foundation with high weights for ASR; Stage 2 introduces complex tasks like multi-talker ASR, code-switching, and timestamp prediction while reducing weights of foundation data to prevent catastrophic forgetting; Stage 3 focuses on domain-specific and specialized data. A key technical contribution is the use of task-aware and dataset-aware weighting to balance the disparate token budgets of generative tasks (TTS/S2ST) versus discriminative tasks (ASR), preventing generative tasks from dominating the loss. For speech generation, the model predicts low-bitrate discrete tokens (325 bps) using a VQ tokenizer trained on W2v-BERT features, which are then decoded into waveforms by a GAN-based vocoder conditioned on source speaker features for identity preservation.
Results & evaluation
The experimental evaluation is extensive, covering over 20 tasks and 100 languages. Walrus demonstrates state-of-the-art performance on multilingual ASR (VoxPopuli, FLEURS), code-switching ASR (CS-FLEURS), and multi-talker ASR (LibriMix, Daily Talk), significantly outperforming comparable open models and even matching or surpassing the much larger Qwen3-Omni (30B) on several benchmarks. Notably, it achieves a cpWER of 19.32 on LibriMix clean 2-mix, which is half the error rate of Qwen3-Omni. The paper also evaluates speech translation (S2ST) and audio understanding (MMSU, MMAU-Pro), showing strong results. The comparison between the Commercial-Ready (CR) and Research (R) variants provides insight into the impact of data licensing and quality, with the R variant showing consistent improvements across all benchmarks.
Reproducibility
The paper provides a high level of detail regarding the training setup, including optimizer hyperparameters, batch sizes, and the specific data weighting schemes for each stage. It lists the specific public datasets used (CommonVoice, VoxPopuli, LibriSpeech, etc.) and describes the synthetic data generation pipelines for code-switching and multi-talker scenarios. However, as is common with large-scale industrial models, the exact composition of the "internal data" and the specific implementation details of the proprietary data cleaning pipelines are not fully disclosed. The code and model weights are available via the provided GitHub link, which enhances reproducibility for those with access to the resources.
Limitations
The primary limitation is the reliance on a fixed backbone (Gemma 4 E4B), which may limit the model's scalability compared to architectures designed specifically for multimodal alignment. The paper does not provide a detailed ablation study isolating the contribution of each individual stage of the curriculum or the specific impact of the task-weighting strategy versus the curriculum order alone. Additionally, the evaluation of audio understanding is somewhat limited compared to the extensive ASR and translation results, and there is no subjective evaluation (e.g., MOS) for the generated speech, relying instead on objective metrics and comparisons to baselines.
Broader impact
This work has significant implications for the deployment of unified speech models in real-world applications, demonstrating that a single model can handle a wide range of tasks (transcription, translation, generation, understanding) efficiently. The emphasis on data engineering provides a roadmap for other researchers and companies to improve their speech models without necessarily increasing model size or complexity. The availability of a commercial-ready variant trained on licensed data is particularly valuable for industry adoption. The results suggest that careful data curation and mixing can yield performance gains comparable to or exceeding those from architectural changes, potentially shifting research focus towards data-centric methods in speech AI. The main contribution is a data-centric training recipe for a unified 8B speech-and-audio foundation model that achieves state-of-the-art results across diverse tasks without architectural novelty. By fixing the backbone and focusing on a three-stage data curriculum with task-aware weighting, the paper demonstrates that data engineering is a primary driver of quality in multi-task speech modeling, outperforming larger models and dedicated task-specific systems on key benchmarks for ASR, translation, and audio understanding.
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform…
Abstract
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral $v$-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen
AI-generated analysis. Check claims against the paper.
Summary
WaveGen proposes a fully end-to-end text-to-waveform diffusion model using Spectral Forcing to jointly learn internal spectral and waveform trajectories, achieving state-of-the-art zero-shot TTS performance without external vocoders or duration predictors. The paper provides rigorous ablation studies on prediction/loss spaces and scalability, demonstrating that direct waveform generation can match or exceed multi-stage pipelines when properly optimized with decoupled $x$-prediction and $v$-loss objectives.
Methodology
The paper proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework. The core methodological contribution is "Spectral Forcing," which jointly learns an internal low-resolution spectral trajectory (Mel-spectrogram) and the target high-resolution waveform trajectory within a single DiT architecture. This approach avoids the need for external vocoders, codecs, or duration predictors. The model utilizes a decoupled prediction and loss space strategy: it predicts the clean signal ($x$-prediction) to ensure accurate energy distribution but computes the loss in the velocity space ($v$-loss) to maintain training stability. Additionally, a multi-scale spectral $v$-loss is introduced to capture time-frequency structures at various resolutions. The architecture employs shared DiT blocks for alignment and disentangled heads for representation-specific refinement.
Results & evaluation
The experiments are extensive, covering scalability (0.1B to 0.6B parameters), dataset scale (LibriTTS to Emilia 120k hours), and ablation studies on key components (Spectral Forcing, prediction/loss spaces, disentangled heads). The model achieves state-of-the-art results on the Seed-en benchmark with a WER of 1.47 and SIM of 0.75, outperforming complex multi-stage pipelines. Subjective evaluations (MOS) via MTurk confirm comparable naturalness and similarity to strong baselines like F5-TTS and CosyVoice3. The paper also provides insights into NFE (Number of Function Evaluations) and CFG (Classifier-Free Guidance) effects.
Reproducibility
The authors provide source code and checkpoints on GitHub. Detailed hyperparameters, training configurations, and implementation details are included in the appendix. The use of open-source datasets (LibriTTS, Emilia) and standard evaluation metrics (Whisper for WER, WavLM for SIM, DNSMOS) enhances reproducibility.
Limitations
The model is limited to 24 kHz sampling rate, whereas state-of-the-art systems often support 48 kHz. The training was restricted to 4 epochs for most models due to computational constraints, though trends suggest further improvement with longer training. The reliance on MP3-compressed Emilia data may limit high-frequency detail generation compared to uncompressed data.
Broader impact
This work demonstrates that fully end-to-end waveform generation is a viable alternative to complex cascaded TTS pipelines, potentially simplifying deployment and reducing error propagation from intermediate stages. It contributes to the understanding of diffusion models in high-dimensional audio spaces and offers a scalable framework for future zero-shot TTS research. WaveGen proposes a fully end-to-end text-to-waveform diffusion model using Spectral Forcing to jointly learn internal spectral and waveform trajectories, achieving state-of-the-art zero-shot TTS performance without external vocoders or duration predictors. The paper provides rigorous ablation studies on prediction/loss spaces and scalability, demonstrating that direct waveform generation can match or exceed multi-stage pipelines when properly optimized with decoupled $x$-prediction and $v$-loss objectives.
We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise…
Abstract
We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.
AI-generated analysis. Check claims against the paper.
Summary
AudioCodex introduces a fully causal, continuous audio autoencoder leveraging Mean Flow and adversarial phase recovery to achieve a superior reconstruction-generation trade-off for real-time applications. The paper demonstrates a rigorous methodology for decoupling magnitude and phase modeling, providing a robust framework for streaming generative audio that outperforms existing non-causal baselines in the critical balance of fidelity and generative usability.
Methodology
The paper proposes AudioCodex, a fully causal autoencoder for live audio generation. The core methodological contribution is the decoupling of magnitude and phase modeling. The magnitude autoencoder utilizes a Mean Flow (specifically Improved Mean Flow, iMF) framework to learn a continuous latent representation from log-magnitude spectrograms. This is a novel application of flow-based generative modeling to audio autoencoding, aiming for single-step decoding stability. The architecture employs a causal convolutional patchifier and frequency-wise transformers. A key innovation is the "bounded-noise augmentation" strategy, which synchronizes latent noise with data-space noise levels to regularize the latent space without degrading reconstruction fidelity. Additionally, Representation Alignment (REPA) is used to align intermediate features with a frozen self-supervised encoder (USAD-Small), enhancing the semantic structure of the latents for downstream generation. The phase is recovered separately using an adversarial estimator (R3GAN-based), which allows the magnitude model to focus on spectral structure without the instability introduced by phase prediction.
Results & evaluation
The experimental section is rigorous, featuring extensive ablations on both the magnitude autoencoder and the phase estimator. The authors evaluate the reconstruction-generation trade-off using Fréchet Audio Distance (FAD) for both reconstruction (rFAD) and downstream generation (gFAD) with a lightweight proxy model. They compare against strong baselines including M2L, CDC, SAO, and DAC. The results demonstrate that AudioCodex achieves a superior balance between reconstruction quality and generative usability compared to non-causal baselines, while maintaining strict causality. The inference speed analysis confirms real-time capability on a single A100 GPU, which is critical for the stated application domain. The use of multiple audio domains (speech, music, general audio) strengthens the generalizability claims.
Reproducibility
The paper provides detailed implementation specifics, including hyperparameters for training (learning rates, batch sizes, optimizer settings), architectural details (channel counts, layer types, downsampling strides), and data preprocessing steps. The release of evaluation metadata and the companion website with audio samples further supports reproducibility. The code and model weights are implied to be available via the project page, though the text does not explicitly state a GitHub link, the companion site is provided.
Limitations
The primary limitation acknowledged is that non-causal decoding still yields better quality than causal decoding, indicating a performance ceiling imposed by the causality constraint. The downstream generation evaluation uses short 1.5-second samples, which may not fully capture long-range musical or speech structure. The reliance on a separate phase estimator adds architectural complexity and potential points of failure in the reconstruction pipeline.
Broader impact
This work is significant for the development of real-time interactive audio systems, such as live music accompaniment, conversational AI with low-latency voice, and interactive sound design tools. By providing a high-quality, streamable continuous latent space, it enables new classes of generative applications that were previously hindered by the latency of non-causal models or the discrete nature of token-based codecs. It bridges the gap between high-fidelity offline audio generation and real-time streaming constraints. AudioCodex introduces a fully causal, continuous audio autoencoder leveraging Mean Flow and adversarial phase recovery to achieve a superior reconstruction-generation trade-off for real-time applications. The paper demonstrates a rigorous methodology for decoupling magnitude and phase modeling, providing a robust framework for streaming generative audio that outperforms existing non-causal baselines in the critical balance of fidelity and generative usability.
Streaming Sortformer introduced an arrival-order speaker cache to maintain speaker identities across chunks in end-to-end streaming speaker diarization, with a single model supporting configurable latency profiles spanning offline operation to ultra-low-latency streaming…
Abstract
Streaming Sortformer introduced an arrival-order speaker cache to maintain speaker identities across chunks in end-to-end streaming speaker diarization, with a single model supporting configurable latency profiles spanning offline operation to ultra-low-latency streaming. Nemotron 3 Diarization addresses the original model's four-speaker limitation by extending support to eight speakers while improving both accuracy and computational efficiency. The previous FastConformer-Transformer stack is streamlined into a pure Transformer encoder. A learned upsampling layer then maps the encoder's coarse 80-ms representations to 10-ms resolution, enabling fine-grained predictions. Training leverages large-scale simulated mixtures and adds activity and phantom-speaker losses to the standard Sortformer objectives. Despite doubling speaker capacity over its predecessor, the new architecture reduces training memory by a factor of 2.8, makes training $2.2\times$ faster, and delivers up to $6.4\times$ higher inference throughput. Nemotron 3 Diarization establishes a new state of the art in DER on DIHARD III and CALLHOME. When used to condition DiCoW, it also achieves state-of-the-art tcpWER on the single-microphone conditions of AMI and NOTSOFAR-1. Training code is open source, and model weights are released under the permissive OpenMDW-1.1 license.
AI-generated analysis. Check claims against the paper.
Summary
Extends the Streaming Sortformer framework to support eight speakers with improved accuracy and efficiency. The paper presents a robust engineering and methodological contribution to the field of speaker diarization, effectively balancing the trade-offs between speaker capacity, temporal resolution, and computational cost through architectural simplification and targeted auxiliary losses.
Methodology
The paper proposes a streamlined architecture for streaming speaker diarization, replacing the previous FastConformer-Transformer stack with a pure 31-layer Transformer encoder. Key methodological contributions include a learned subpixel convolution layer for upsampling 80-ms representations to 10-ms resolution, which allows for fine-grained boundary detection without increasing encoder computational cost. The authors introduce two auxiliary losses: an activity loss to model overall speech activity (silence, single, overlap) and a phantom loss to suppress false activations in unused speaker channels, addressing the overcounting issue inherent in fixed-capacity EEND models. The training regimen involves a two-stage process: offline pre-training on simulated mixtures initialized from a self-supervised NEST model, followed by streaming fine-tuning with an Arrival-Order Speaker Cache (AOSC) and block-causal tail augmentation to simulate varying latency contexts.
Results & evaluation
The experimental evaluation is extensive, covering both offline and streaming settings across multiple latency profiles (30.4s, 1.04s, 0.32s). The model achieves state-of-the-art DER on DIHARD III and CALLHOME, and SOTA tcpWER on AMI and NOTSOFAR-1 when conditioning DiCoW. Ablation studies rigorously quantify the impact of the backbone change, upsampling layer, auxiliary losses, and data composition. The paper also provides strong evidence of efficiency gains, reporting a 2.8x reduction in training memory and 6.4x higher inference throughput compared to the predecessor, despite doubling the speaker capacity.
Reproducibility
Reproducibility is high. The authors release the model weights under a permissive license and provide the training code via the NVIDIA NeMo Speech Framework. Specific configuration files for both offline and streaming training are linked in the paper. Detailed descriptions of data sources, simulation protocols (FastMSS), and training hyperparameters are provided.
Limitations
The model is still limited to a maximum of 8 speakers, which may be insufficient for very large group conversations. The evaluation relies heavily on simulated mixtures for training, which may introduce domain gaps not fully captured by the augmentation strategies. The paper notes that DER comparisons on some datasets (AMI, AliMeeting, NOTSOFAR-1) are not directly comparable to previous SOTA due to differences in ground-truth annotation styles (forced-aligned vs. segment-level), although they mitigate this by also reporting tcpWER.
Broader impact
This work significantly advances the practicality of streaming speaker diarization by making high-accuracy, multi-speaker models computationally efficient enough for real-time deployment on consumer hardware. The open-sourcing of weights and code under a permissive license facilitates widespread adoption in both research and industry, particularly for meeting transcription and multi-talker ASR systems. The integration with downstream ASR models like DiCoW demonstrates the utility of the diarization component in end-to-end speech processing pipelines. Extends the Streaming Sortformer framework to support eight speakers with improved accuracy and efficiency. The paper presents a robust engineering and methodological contribution to the field of speaker diarization, effectively balancing the trade-offs between speaker capacity, temporal resolution, and computational cost through architectural simplification and targeted auxiliary losses.
As modern text-to-speech systems become increasingly intelligible and natural, evaluation is shifting from detecting gross synthesis failures to identifying subtle departures from lifelike human delivery. Because large-scale listening tests are costly and slow, speech reward…
Abstract
As modern text-to-speech systems become increasingly intelligible and natural, evaluation is shifting from detecting gross synthesis failures to identifying subtle departures from lifelike human delivery. Because large-scale listening tests are costly and slow, speech reward models can serve as scalable automatic judges for comparing generated utterances, curating data, and supporting iterative TTS development. Existing speech reward models can predict scores or pairwise preferences and may generate natural-language rationales, but their supervision is typically defined at the utterance or pair level. The connection between local acoustic phenomena and the final preference therefore remains implicit. We introduce REAL, a framework for Reward modeling through Evidence-linked Analysis of Lifelikeness in generated speech. Its central idea is to supervise not only the final pairwise preference but also a structured account of its supporting observations, linking local audio spans to perceptual phenomena, affected dimensions, and the overall judgment. Specifically, REAL-Jury constructs these evidence-linked reports through timestamp alignment, multi-model analysis, and dispute-oriented deliberation. REAL-Judge learns to generate them from paired audio inputs. REAL-Eval assesses preference prediction in two tiers, from salient perceptual differences to subtle distinctions among strong TTS systems. Instantiated with MiMo-Audio, REAL-Judge improves pairwise preference accuracy over its backbone by 16.01 and 11.41 percentage points on the two tiers, respectively, and outperforms the open-source speech reward models SpeechJudge and UniSRM on both tiers.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a novel framework for speech reward modeling that explicitly links local acoustic evidence to global preference judgments through a multi-model deliberation process. By constructing structured, timestamp-aligned supervision signals and distilling them into a single model, REAL-Judge achieves state-of-the-art performance in both salient defect detection and subtle preference discrimination, offering a scalable and interpretable alternative to human evaluation for TTS systems.
Methodology
The paper proposes REAL, a framework for speech reward modeling that explicitly supervises the link between local acoustic evidence and global preference judgments. The methodology is structured into three components: REAL-Jury, which uses a multi-model orchestration (coordinator + specialized models) to generate timestamp-aligned, dispute-resolved evidence reports; REAL-Judge, which distills this structured supervision into a single audio-language model using LoRA; and REAL-Eval, a two-tier benchmark. The core technical contribution is the "evidence-linked" supervision signal, which forces the model to ground its preference in specific time spans and perceptual phenomena rather than relying on implicit global features. The use of explicit time-marker embeddings in the audio representation is a practical and effective design choice for temporal grounding. The deliberation process in REAL-Jury is a sophisticated approach to data construction, leveraging model disagreement to refine evidence quality.
Results & evaluation
The experimental setup is robust, utilizing a large training set (11,947 pairs) constructed via the proposed pipeline. The evaluation is well-designed with two distinct tiers: Standard (salient defects) and Fine (subtle differences among strong systems). The results show significant improvements over the backbone model (MiMo-Audio) and outperform strong baselines like SpeechJudge and UniSRM. The ablation studies effectively demonstrate the contribution of time markers and the deliberation process. The correlation analysis (Pearson r=0.96) between predicted and human win rates is a strong indicator of the model's utility for system-level evaluation.
Reproducibility
The paper provides detailed hyperparameters for training (LoRA rank, learning rate, etc.) and describes the architecture of the jury. However, the specific prompts used for the coordination model and the exact criteria for "dispute resolution" are not fully detailed in the provided text, which may limit exact reproduction of the data construction pipeline. The code and weights are promised to be released, which is a positive sign.
Limitations
The work is currently limited to Chinese speech, which restricts its immediate generalizability. The reliance on a complex multi-model jury for data construction makes the training data generation process computationally expensive and potentially difficult to scale to other languages or domains without significant engineering effort. The evaluation, while strong, is based on a relatively small number of pairs (400 and 207) for the fine-grained tier, which may affect statistical significance.
Broader impact
This work has significant potential for automating the evaluation of TTS systems, reducing the need for costly human listening tests. The evidence-linked approach could be extended to other audio tasks, such as music generation evaluation or audio captioning, where grounding in specific temporal segments is crucial. The framework offers a path towards more interpretable and trustworthy AI judges for audio content. The paper introduces a novel framework for speech reward modeling that explicitly links local acoustic evidence to global preference judgments through a multi-model deliberation process. By constructing structured, timestamp-aligned supervision signals and distilling them into a single model, REAL-Judge achieves state-of-the-art performance in both salient defect detection and subtle preference discrimination, offering a scalable and interpretable alternative to human evaluation for TTS systems.
Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this…
Abstract
Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this approach has not yet been extended to audio diffusion models. We introduce MAAS (Multi-Area Auditory Synthesis), an end-to-end framework that conditions a pretrained audio diffusion transformer on ROI-wise fMRI responses without first supervising a decoder to predict an external stimulus representation. MAAS uses region-specific projections and learned brain tokens to condition Stable Audio through a dedicated cross-attention pathway. On a 7T fMRI dataset, we demonstrate consistent improvements over baselines, including two-stage fMRI-to-latent conditioning, across both acoustic and semantic evaluation metrics. In addition, we introduce a fidelity metric grounded in a computational model of auditory neural processing, which allows us to quantify the contribution of individual auditory cortical regions to reconstructed audio representations.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a direct conditioning framework for audio diffusion transformers on fMRI data, demonstrating improved reconstruction fidelity and providing interpretable insights into cortical contributions. The technical contribution is significant in adapting diffusion transformer architectures for non-text, continuous neural signals, and the experimental design, including the novel biologically grounded fidelity metric, is rigorous and well-motivated.
Methodology
The paper proposes MAAS, a framework that directly conditions a pretrained audio diffusion transformer (Stable Audio) on fMRI data, bypassing the traditional two-stage pipeline of decoding neural signals into intermediate embeddings (like CLAP or YamNet) before synthesis. The core technical contribution is an adapter mechanism that injects region-specific fMRI projections into the cross-attention layers of the DiT backbone. This requires handling architectural mismatches, specifically preserving Rotary Positional Embeddings (RoPE) asymmetry and Grouped-Query Attention (GQA) structures while introducing a parallel "brain" branch for key-value projections. The method uses learned brain tokens to aggregate voxel-level activity from six bilateral auditory ROIs, preserving anatomical organization. The approach is sound in its theoretical motivation to avoid information bottlenecks inherent in intermediate representation prediction.
Results & evaluation
The experiments are conducted on a 7T fMRI dataset with 5 participants and 288 natural sounds. The evaluation is rigorous, comparing MAAS against two-stage baselines (Brain-to-text, Brain-only) and oracle caption-based conditions. The paper introduces a novel fidelity metric grounded in the NSL Auditory-Cortical Toolbox to quantify the contribution of individual cortical regions, which is a significant methodological addition. Results show MAAS outperforms baselines in CLAP similarity and pairwise identification. Ablation studies reveal a hierarchical contribution of auditory regions, with early areas (HG, PT) providing more self-sufficient spectrotemporal information. External validation on a 3T dataset further supports generalizability. However, the small sample size (5 participants) is a notable constraint.
Reproducibility
The authors provide code via an anonymous repository and a demo website with audio samples. Hyperparameters, training details (AdamW, cosine annealing, gradient clipping), and dataset preprocessing steps are described in detail. The use of standard libraries (fMRIprep) and open-source backbones (Stable Audio) enhances reproducibility. The specific voxel selection criterion (Pearson r with YamNet) is clearly defined, though it introduces a dependency on YamNet for preprocessing, which is explicitly stated not to be a conditioning signal.
Limitations
The primary limitation is the small dataset size (5 participants, 288 stimuli), which limits statistical power and generalizability claims. The temporal resolution of fMRI (seconds) is inherently low for audio, and the paper acknowledges this, noting that the model relies on spatial patterns rather than fine-grained temporal dynamics. The reliance on a specific 7T dataset for primary results, while external validation is provided, means the method's performance on standard 3T clinical data might vary. The "empty prompt" training strategy, while effective for isolating brain signals, may limit the model's ability to leverage semantic text cues if available in future applications.
Broader impact
This work bridges the gap between neuroscience and generative AI, offering a new paradigm for brain-to-sound reconstruction. The direct conditioning approach could be extended to other sensory modalities or different generative architectures. The interpretability provided by the ROI-level ablation analysis offers insights into the functional organization of the auditory cortex, potentially benefiting both computational neuroscience and the development of auditory brain-computer interfaces for clinical applications. The paper introduces a direct conditioning framework for audio diffusion transformers on fMRI data, demonstrating improved reconstruction fidelity and providing interpretable insights into cortical contributions. The technical contribution is significant in adapting diffusion transformer architectures for non-text, continuous neural signals, and the experimental design, including the novel biologically grounded fidelity metric, is rigorous and well-motivated.
Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio…
Abstract
Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces LAION-Audio-XL, a 365M audio-text dataset, and CLAP2, a family of contrastive audio-language models, demonstrating that data scaling is more effective than model scaling for contrastive pretraining within the tested compute budget. The comprehensive data curation pipeline and rigorous scaling analysis provide valuable insights for the field, while the release of the dataset and models facilitates future research.
Methodology
The paper presents a robust and well-documented methodology for constructing a large-scale audio-text dataset (LAION-Audio-XL) and training contrastive models (CLAP2). The data pipeline is extensive, involving multi-stage curation including exact and near-duplicate removal (using Olaf fingerprints and SemDeDup), signal quality checks, and a sophisticated taxonomy-based selection process to ensure diversity. The use of a 4,416-node taxonomy for coverage-aware selection is a strong methodological contribution. The model architecture utilizes a NaFlex audio encoder and a ModernBERT-based text encoder, trained from scratch to avoid confounding effects from pretraining. The scaling study methodology is rigorous, employing warmup-stable-decay learning rate schedules and fitting power laws to downstream error rather than just validation loss, which provides more practical insights for compute allocation.
Results & evaluation
The experimental evaluation is comprehensive, covering five model sizes (27M to 340M parameters) and eleven data scales (1M to 1024M samples). The results on MAEB-30, HEAR, and MSEB benchmarks show that the models outperform prior audio-text models of comparable size. The scaling analysis reveals that data scaling is more effective than model scaling for contrastive audio-language pretraining within the tested range, with optimal compute allocation favoring more samples over larger models. The ablation studies on audio length (30s vs 120s) and caption length provide useful practical insights. The comparison with larger models (1B+ parameters) demonstrates the efficiency of the proposed approach.
Reproducibility
The paper is highly reproducible. The authors release the dataset, captions, model checkpoints, and code. The data pipeline is described in detail in the appendix, including specific thresholds for deduplication and selection. The training hyperparameters are provided, and the use of standard libraries (timm, faiss, etc.) facilitates implementation. The release of the 365M sample dataset is a significant contribution to the community, enabling further research on scaling laws and data quality.
Limitations
The primary limitation is the reliance on web-scraped data, which may contain biases and noise despite the extensive curation pipeline. The models are evaluated primarily on classification and retrieval tasks, with limited evaluation on generative tasks or other downstream applications. The scaling study is limited to models up to 340M parameters and data up to 1B samples, so the behavior at larger scales remains unknown. The use of model-generated captions for a large portion of the dataset may introduce captioning errors that affect model performance.
Broader impact
The release of LAION-Audio-XL and CLAP2 models will significantly impact the audio-language research community by providing a large, high-quality dataset and strong baseline models. The scaling insights will guide future research in compute allocation for audio pretraining. The dataset's diversity and size enable research on long-tail sound events and multilingual audio understanding. The open-source nature of the work promotes transparency and reproducibility in audio-language learning. The paper introduces LAION-Audio-XL, a 365M audio-text dataset, and CLAP2, a family of contrastive audio-language models, demonstrating that data scaling is more effective than model scaling for contrastive pretraining within the tested compute budget. The comprehensive data curation pipeline and rigorous scaling analysis provide valuable insights for the field, while the release of the dataset and models facilitates future research.
We propose a music inpainting technique combining similarity matching and a diffusion-based generative model to replace large segments of audio recordings with suitable alternatives. The proposed hybrid method, Similarity-Guided Diffusion Posterior Sampling (SimDPS), retrieves…
Abstract
We propose a music inpainting technique combining similarity matching and a diffusion-based generative model to replace large segments of audio recordings with suitable alternatives. The proposed hybrid method, Similarity-Guided Diffusion Posterior Sampling (SimDPS), retrieves candidate replacement segments from the wider signal based on contextual similarity and uses them to guide the diffusion process toward coherent reconstructions. Being model-agnostic, the framework is compatible with any pre-trained diffusion model operating in either the waveform or latent domain. Objective evaluations demonstrate that SimDPS outperforms widely adopted RePaint and DPS methods across acoustic features and neural-embedding distance metrics. Subjective evaluations on multi-genre music datasets confirm that SimDPS enhances perceptual plausibility compared to previous diffusion-based methods and demonstrates robustness to poorly matched similarity candidates. These findings pave the way for improved music restoration and context-aware audio editing.
AI-generated analysis. Check claims against the paper.
Summary
[The paper introduces SimDPS, a training-free, model-agnostic framework that enhances long-gap music inpainting by guiding diffusion posterior sampling with similarity-retrieved candidates, effectively combining the structural reliability of retrieval with the generative flexibility of diffusion models to improve perceptual plausibility and musical coherence.].
Methodology
The paper proposes SimDPS, a hybrid framework for long-gap music inpainting that integrates similarity-based retrieval with diffusion posterior sampling (DPS). The core innovation lies in the "Similarity-Guided" mechanism, where retrieved candidate segments from the source signal act as an auxiliary noisy observation in the missing region. This allows the diffusion model to be guided by structural priors from the track itself, mitigating the "identity drift" common in pure generative approaches for long gaps. The method is model-agnostic, supporting both latent (ACE-Step) and waveform (MR-CQTdiff) domains. The introduction of a multi-candidate ensemble (SimDPS-Multi) with a spectral loss function to handle conflicting gradients is a thoughtful technical addition, although its effectiveness is context-dependent.
Results & evaluation
The evaluation is robust, covering both objective metrics (MFCC, Chroma, Rhythm, Boundary Error, MusicFM, CLAP) and subjective listening tests (MUSHRA). The use of MUSDB18-HQ for general music and MAESTRO for piano provides a good breadth of genres. The results clearly show SimDPS outperforming baselines like RePaint, DPS, and pure similarity search, particularly in preserving musical coherence over 5-second gaps. The subjective tests confirm that the method improves perceptual plausibility, especially when the retrieved similarity match is imperfect.
Reproducibility
The paper provides high reproducibility. It specifies the exact models used (ACE-Step 1.5, MR-CQTdiff), hyperparameters (noise schedules, guidance weights, context lengths), and datasets. Code and audio examples are available via the provided URL. The detailed description of the similarity search procedure and the modified likelihood formulation allows for faithful re-implementation.
Limitations
The method relies on the availability of a source corpus (the rest of the track) for similarity search, which limits its applicability to scenarios where the full track is not available or where the missing segment is unique (no repetition). The subjective evaluation sample size (23 and 17 participants) is modest. The performance of SimDPS-Multi is inconsistent across genres, sometimes performing worse than SimDPS-Solo or even baselines when the retrieved matches are poor or conflicting.
Broader impact
This work contributes to the field of audio restoration and editing by providing a practical, training-free method for long-gap inpainting. It bridges the gap between deterministic retrieval methods and stochastic generative models, offering a path toward more robust audio editing tools. The model-agnostic nature of the framework makes it applicable to future diffusion-based audio models. [The paper introduces SimDPS, a training-free, model-agnostic framework that enhances long-gap music inpainting by guiding diffusion posterior sampling with similarity-retrieved candidates, effectively combining the structural reliability of retrieval with the generative flexibility of diffusion models to improve perceptual plausibility and musical coherence.].
Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending…
Abstract
Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending hearing to sounds that are inaudible to humans. More broadly, the physical world is full of waves, vibrations, and oscillations, only a small fraction of which are directly audible to humans. This invites the extension of multimodal LLMs toward a broader form of machine hearing, which remains largely underexplored. To investigate this direction, we introduce the Machine Hearing Benchmark (MHB), comprising 6,128 instances across 13 tasks and seven domains. MHB spans highly heterogeneous input regimes, with sampling rates from approximately $10^{-5}$ to $10^6$ Hz, sequence lengths from $10^1$ to $10^6$ samples, and both single- and multi-channel inputs. Evaluations of current open-source and proprietary Audio LLMs (ALLMs) reveal limited zero-shot transfer to machine hearing. Supervised adaptation on machine-hearing data alone does not close this gap, suggesting that the challenge extends beyond data exposure to a systematic mismatch between existing audio front ends and machine-hearing inputs. To address this mismatch, we introduce MachEar, a lightweight encoder extension that complements the pretrained Audio LLM for machine-hearing inputs. The results show that MachEar equips a pretrained Audio LLM with stronger machine-hearing capabilities while largely preserving its existing audio understanding performance.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces the Machine Hearing Benchmark and the MachEar encoder extension, demonstrating that current Audio LLMs struggle with non-human-audible signals and proposing a lightweight architectural fix to bridge this gap. The technical contribution is strong, combining a well-curated benchmark with a targeted architectural innovation. The methodology is sound, using parallel processing to handle heterogeneous signal regimes without disrupting the pretrained LLM's capabilities. The significance to the field is high, as it shifts the focus from human-centric audio understanding to broader physical world perception, addressing a critical limitation in current multimodal models. The rigorous evaluation against both general and domain-specific models provides a clear baseline and validates the proposed solution's effectiveness.
Methodology
The paper proposes a novel paradigm of "Machine Hearing," extending Audio LLMs (ALLMs) beyond human-audible frequencies to process signals ranging from $10^{-5}$ Hz to $10^6$ Hz. The methodology involves constructing the Machine Hearing Benchmark (MHB) with 6,128 instances across 13 tasks in 7 domains (Astronomy, Earth Science, Healthcare, etc.). The core technical contribution is MachEar, a lightweight encoder extension that processes inputs in parallel with the pretrained audio encoder. MachEar utilizes parallel FFT and 1D CNN pathways to extract spectral and temporal features, which are then processed by Transformer blocks to generate compact tokens. This design aims to address the systematic mismatch between standard audio front-ends (optimized for 16-48kHz speech/audio) and extreme machine-hearing inputs (ultra-low or ultra-high frequency, short sequences). The approach is elegant in its modularity, allowing it to complement rather than replace the existing LLM backbone.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors benchmark seven state-of-the-art ALLMs (including Qwen2.5-Omni, Audio-Flamingo-3, Gemini-3-Flash, GPT-Audio-Mini, and Doubao) in a zero-shot setting. The results clearly demonstrate that current ALLMs fail significantly on machine-hearing tasks, often performing below random guessing. The paper also evaluates domain-specific foundation models (e.g., SeisLM, SpectrumFM) to provide context, showing that while domain pretraining helps, it doesn't solve the generalization issue for ALLMs. The introduction of MachEar shows substantial improvements in Macro-F1 scores for Qwen2.5-Omni, validating the hypothesis that the bottleneck lies in the audio front-end processing rather than just the LLM's reasoning capabilities. The analysis of failure modes (e.g., short-input failures, unrecognized inputs) provides valuable diagnostic insights.
Reproducibility
The paper provides high reproducibility. The benchmark is released on HuggingFace, and the code for MachEar is presumably available (linked in the project URL). The experimental setup, including model versions, prompting protocols, and evaluation metrics (Macro-F1, Success Rate), is clearly defined. The use of public datasets and standard models further enhances reproducibility.
Limitations
The primary limitation is the reliance on a specific base model (Qwen2.5-Omni) for the MachEar demonstration; while the architecture is general, the specific gains might vary across different LLM backbones. Additionally, the benchmark, while diverse, is relatively small (6,128 instances) for such a broad range of domains, which may limit the statistical robustness of some conclusions. The paper focuses on classification tasks; the ability to perform generative or reasoning tasks on these signals is not deeply explored.
Broader impact
This work has significant broader impact by defining a new frontier for multimodal AI: perception beyond human sensory limits. It opens avenues for AI in scientific discovery (astronomy, geophysics), industrial monitoring, and healthcare diagnostics where signals are not human-audible. The proposed MachEar module is a practical solution that can be integrated into existing ALLMs, potentially accelerating the adoption of AI in these specialized fields. The paper introduces the Machine Hearing Benchmark and the MachEar encoder extension, demonstrating that current Audio LLMs struggle with non-human-audible signals and proposing a lightweight architectural fix to bridge this gap. The technical contribution is strong, combining a well-curated benchmark with a targeted architectural innovation. The methodology is sound, using parallel processing to handle heterogeneous signal regimes without disrupting the pretrained LLM's capabilities. The significance to the field is high, as it shifts the focus from human-centric audio understanding to broader physical world perception, addressing a critical limitation in current multimodal models. The rigorous evaluation against both general and domain-specific models provides a clear baseline and validates the proposed solution's effectiveness.
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently…
Abstract
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces DiCoP and Pandemonium, which integrate diarization conditioning into a long-form Parakeet backbone with cross-speaker attention to improve target-speaker ASR in multi-talker scenarios. This work addresses significant structural limitations of previous diarization-conditioned ASR systems by enabling joint optimization and cross-speaker interaction modeling, supported by a robust pseudo-labeling pipeline for data scaling, resulting in state-of-the-art performance on meeting benchmarks.
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision…
Abstract
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior self-supervised audio embedding models on clip-level classification tasks spanning environmental sound, speech, and music, and on frame-level sound event detection. All code and data are available at https://github.com/labhamlet/AudioSphere and https://huggingface.co/labhamlet.
AI-generated analysis. Check claims against the paper.
Summary
AudioSphere introduces a multi-channel masked autoencoder with joint spatiotemporal masking to learn self-supervised spatial audio representations, achieving state-of-the-art performance on both real-world SELD and monaural audio tasks. The paper makes a significant technical contribution by bridging the gap between monaural SSL and spatial audio modeling, demonstrating that spatial cues can be learned effectively without explicit supervision and that such representations transfer robustly to a wide range of downstream tasks. The introduction of the RealSELD benchmark and the open-sourcing of the model and data pipeline further enhance the paper's impact, providing the community with essential tools for advancing spatial audio research.
Methodology
The paper proposes AudioSphere, a multi-channel masked autoencoder (MAE) designed for self-supervised learning of spatial audio representations. The core innovation lies in the joint modeling of acoustic content (log-mel spectrograms) and spatial cues (intensity vectors) derived from First-Order Ambisonics (FOA). The authors introduce a "tube masking" strategy that masks time-frequency patches across all channels simultaneously, preventing the model from trivially copying information from unmasked channels at the same spatial location. This forces the encoder to learn correlations between acoustic features and directional cues. The architecture utilizes a ViT-Base encoder and a decoder with local-global attention. A significant methodological contribution is the data generation pipeline, which synthesizes 70,000 Ambisonic room impulse responses (ARIRs) using Matterport3D meshes and Monte Carlo ray tracing, paired with online mixing of AudioSet clips. This addresses the scarcity of large-scale, labeled spatial audio data. The use of Ambisonics Signal Matching (ASM) allows the model to generalize across different microphone array geometries (tetrahedral, planar, spherical) without retraining, a robust engineering choice that enhances utility.
Results & evaluation
The experimental evaluation is comprehensive, addressing both the transferability of spatial features to monaural tasks and the efficacy of the model for spatial tasks. The authors introduce RealSELD, a standardized benchmark for real-world SELD, which is a valuable contribution to the field. Results show that AudioSphere, when used as a frozen backbone with shallow MLP probes, outperforms fully supervised baselines on static SELD tasks and transfers competitively to dynamic environments. Crucially, the paper demonstrates that spatial pre-training does not degrade performance on monaural tasks; in fact, it matches or exceeds prior state-of-the-art monaural SSL models (like MW-MAE) on the HEAR benchmark suite, which includes environmental sound, speech, and music classification. The ablation studies on masking strategies (tube vs. per-channel) provide strong evidence for the necessity of the proposed joint masking approach. The comparison with MUSIC and Binaural Signal Matching pipelines on real-world data further validates the practical superiority of the learned representations.
Reproducibility
The authors provide high reproducibility by releasing all code, data, and model weights on GitHub and HuggingFace. The detailed description of the simulation pipeline, including the specific parameters for Matterport3D scenes and the online mixing process, allows other researchers to replicate the training data generation. The use of standard components (ViT, ASM) and clear architectural specifications further supports reproducibility. The release of the RealSELD benchmark kit is particularly beneficial for the community, enabling standardized evaluation of future spatial audio models.
Limitations
A primary limitation is the reliance on simulated room impulse responses for pre-training. While the authors argue that the model generalizes to real-world recordings, the gap between simulated and real-world acoustic conditions (e.g., non-linearities, complex material interactions) may still exist, though the results on RealSELD suggest this gap is manageable. The model is trained on First-Order Ambisonics, which limits the spatial resolution to 4 channels; higher-order Ambisonics could provide finer directional detail but would increase computational cost. Additionally, the pre-training corpus, while large (4.8k hours of AudioSet mixed with simulations), is still smaller than the massive corpora used for some monaural models (e.g., Dasheng's 272k hours), which might limit the ceiling of performance on certain acoustic tasks.
Broader impact
This work has significant implications for the field of spatial audio understanding. By demonstrating that self-supervised learning can effectively capture joint acoustic-spatial features, it opens the door to scalable, label-free training for spatial audio models. The release of RealSELD provides a much-needed standardized benchmark for evaluating real-world SELD, addressing a long-standing issue of fragmented evaluation in this subfield. The ability to use a frozen backbone for both monaural and spatial tasks suggests that spatial pre-training can be a general-purpose enhancement for audio representation learning, potentially benefiting applications in augmented reality, virtual reality, and assistive technologies for the deaf and hard of hearing. AudioSphere introduces a multi-channel masked autoencoder with joint spatiotemporal masking to learn self-supervised spatial audio representations, achieving state-of-the-art performance on both real-world SELD and monaural audio tasks. The paper makes a significant technical contribution by bridging the gap between monaural SSL and spatial audio modeling, demonstrating that spatial cues can be learned effectively without explicit supervision and that such representations transfer robustly to a wide range of downstream tasks. The introduction of the RealSELD benchmark and the open-sourcing of the model and data pipeline further enhance the paper's impact, providing the community with essential tools for advancing spatial audio research.
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text…
Abstract
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces IRIS-Embed, a novel method for instruction-aware speech retrieval that adapts Large Audio-Language Models by learning instruction-conditioned weights to fuse hidden states from different audio encoder layers. This approach significantly outperforms existing baselines on the INSPIRE benchmark and demonstrates strong generalization to broader audio embedding tasks, offering a scalable and effective framework for context-aware audio retrieval.
Methodology
The paper proposes IRIS-Embed, a framework for adapting Large Audio-Language Models (LALMs) into embedding models for instruction-aware speech retrieval. The core methodological contribution is "instruction-aware audio layer weighting." The authors observe that different retrieval tasks (e.g., speaker matching vs. content retrieval) benefit from different layers of the frozen audio encoder. To exploit this, they introduce a lightweight module that takes the text instruction as input and predicts softmax weights to combine hidden states from all layers of the audio encoder. This weighted representation is then projected and fed into the language model (adapted via LoRA) to generate the final embedding. The training objective is InfoNCE contrastive loss, with careful handling of false negatives by masking potentially relevant documents from the batch pool. The data construction pipeline is robust, combining real data from various corpora (VoxCeleb, UrbanSound8K, etc.) with synthetic data generated via TTS to control specific attributes like speaker identity and style.
Results & evaluation
The experiments are extensive and well-controlled. The primary evaluation is on the INSPIRE benchmark, which is specifically designed for instruction-aware speech retrieval. The model significantly outperforms strong baselines, including specialized encoders (ECAPA-TDNN, emotion2vec) and other LALM-based embeddings (LCO-Embedding). The best model achieves 62.92% Recall@50, surpassing the strongest baseline (45.15%) by a large margin. Ablation studies confirm the effectiveness of both instruction awareness and the audio layer weighting mechanism. Furthermore, the model's generalization is tested on the MAEB benchmark, showing consistent gains across 15 diverse audio tasks, indicating that the learned embeddings are not overfitted to the specific retrieval task but capture broad audio-linguistic alignment.
Reproducibility
The authors provide a public GitHub repository containing code and data construction scripts. The paper details the training hyperparameters, data sources, and specific model architectures (Qwen2.5-Omni-3B, Gemma-4-E2B-it). The use of standard components (LoRA, InfoNCE, frozen encoders) and open-source models enhances reproducibility. The detailed description of the data construction pipeline, including how synthetic data is generated and how instructions are formulated, allows other researchers to replicate the dataset.
Limitations
One potential limitation is the reliance on a specific set of LALMs (Qwen2.5-Omni and Gemma-4). While these are state-of-the-art, the method's performance on other architectures or smaller models might vary. Additionally, the instruction-aware weighting module adds computational overhead during inference, as it requires processing the instruction to generate layer weights, although this is likely negligible compared to the LLM inference. The paper focuses on English speech; performance on multilingual or low-resource languages is not evaluated.
Broader impact
This work bridges the gap between general audio-language understanding and specific retrieval tasks. By demonstrating that LALMs can be effectively adapted for retrieval via layer weighting, it opens avenues for more flexible and context-aware audio search systems. The approach could be extended to other multimodal retrieval tasks (video, image) where different modalities or tasks might benefit from different feature layers. The public release of the dataset and code will likely accelerate research in instruction-aware retrieval. The paper introduces IRIS-Embed, a novel method for instruction-aware speech retrieval that adapts Large Audio-Language Models by learning instruction-conditioned weights to fuse hidden states from different audio encoder layers. This approach significantly outperforms existing baselines on the INSPIRE benchmark and demonstrates strong generalization to broader audio embedding tasks, offering a scalable and effective framework for context-aware audio retrieval.
Music source separation extracts vocals and instruments from a mixed recording, enabling remixing, content production, and music analysis. Existing systems commonly operate on waveforms or full-band spectrograms. Waveform inputs contain many samples even in short segments…
Abstract
Music source separation extracts vocals and instruments from a mixed recording, enabling remixing, content production, and music analysis. Existing systems commonly operate on waveforms or full-band spectrograms. Waveform inputs contain many samples even in short segments, whereas a single spectrogram resolution imposes a shared time-frequency resolution trade-off across frequency ranges. We propose SubRoFormer, a Transformer-based system comprising a subband module and a residual RoFormer module. The subband module decomposes the mixture into frequency-localized, lower-rate signals for separation. It then reconstructs the target waveforms from the estimated subbands. The residual RoFormer module models temporal and cross-band relationships on temporally compressed features. It combines this context with fine-rate features through gated residual fusion to estimate source-specific complex masks. On the MUSDB18-HQ test set, training only on MUSDB18-HQ yields an average signal-to-distortion ratio (SDR) of 10.63 dB across four stems using target-specific models, achieving state-of-the-art performance among the compared systems without extra training data. With 500 additional training songs, a single model jointly separating all four stems achieves an average SDR of 11.59 dB, establishing state-of-the-art performance among the compared single-model multi-stem systems trained with extra data. Code and checkpoints are released at https://github.com/ArchiMickey/SubRoformer.
AI-generated analysis. Check claims against the paper.
Summary
SubRoFormer introduces a subband time-domain representation that effectively resolves the time-frequency resolution trade-off in music source separation, achieving state-of-the-art performance on MUSDB18-HQ with improved parameter efficiency. The paper demonstrates a rigorous integration of signal processing principles with Transformer-based architectures, providing a robust and reproducible framework for high-fidelity audio separation.
Methodology
The paper proposes SubRoFormer, a hybrid architecture that combines a fixed subband time-domain frontend with a residual RoFormer separator. The frontend decomposes the mixture into frequency-localized, lower-rate complex baseband signals using a triangular filter bank, demodulation, and decimation. This addresses the resolution trade-off of full-band STFTs by allowing different frequency resolutions. The separator operates on local STFTs of these subbands, using a temporally patched axial RoFormer to model long-range dependencies while retaining fine-rate features via a skip connection. A GEGLU fusion module combines the coarse-rate context with fine-rate features to predict complex masks. The method is mathematically sound, leveraging standard signal processing principles (Nyquist, polyphase implementation) to ensure efficient and near-perfect reconstruction of the subbands before separation. The use of a fixed, non-learned frontend is a deliberate design choice to isolate the separation task from representation learning, which is a valid and rigorous approach.
Results & evaluation
The experiments are conducted on the MUSDB18-HQ dataset, the standard benchmark for music source separation. The paper reports state-of-the-art results, achieving 10.63 dB average SDR with target-specific models and 11.59 dB with a single multi-stem model using extra data. The ablation studies are thorough, examining the impact of the subband frontend, temporal patching, and fusion mechanisms. The paper also provides a detailed analysis of the reconstruction quality of the subband frontend, showing it introduces minimal distortion compared to the STFT. The comparisons are fair, using the same training data and augmentation policies as baselines where possible. The inclusion of parameter efficiency comparisons is a strong point, showing that the method achieves high performance with fewer parameters than some baselines.
Reproducibility
The paper provides high reproducibility. Code and checkpoints are released on GitHub. The implementation details, including filter bank specifications, training hyperparameters, and augmentation policies, are described in detail. The use of standard libraries (SciPy for FIR filters) and common frameworks (PyTorch implied by the architecture) further enhances reproducibility. The paper clearly distinguishes between the fixed frontend and the learned separator, making it easier to replicate the system.
Limitations
The primary limitation is the computational cost of the subband frontend, although the polyphase implementation mitigates this. The method relies on a fixed filter bank, which may not be optimal for all types of music or sources. The paper does not extensively discuss the impact of the subband count and bandwidth on separation performance, only providing a single ablation with 64 subbands. The use of an internal dataset of 500 songs for the extra-data experiments limits the generalizability of those results, as the dataset is not publicly available. The paper also does not provide subjective listening tests, which are important for evaluating the perceptual quality of separated stems.
Broader impact
The proposed method has significant potential for applications in music production, remixing, and audio restoration. The ability to separate stems with high fidelity and efficiency could enable new tools for musicians and producers. The subband approach could be extended to other audio separation tasks, such as speech enhancement or environmental sound separation. The paper contributes to the broader field of audio signal processing by demonstrating the effectiveness of combining classical signal processing techniques with modern deep learning architectures. SubRoFormer introduces a subband time-domain representation that effectively resolves the time-frequency resolution trade-off in music source separation, achieving state-of-the-art performance on MUSDB18-HQ with improved parameter efficiency. The paper demonstrates a rigorous integration of signal processing principles with Transformer-based architectures, providing a robust and reproducible framework for high-fidelity audio separation.
Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real…
Abstract
Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real workflows where several changes are requested at once. Here, we present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30s music excerpts end-to-end from free-form instructions, including compound requests. ProCraft supports seven operations across eight instruments, from tasks with a unique target (e.g., track removal) to creative ones (e.g., accompaniment generation). We align intermediate representations with pretrained music embeddings and pitch and instrument targets to preserve content outside the edit, and gated cross-attention in every decoder layer to keep generation tied to the instruction. To train on compound requests, we introduce On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes. ProCraft performs competitively against task-specific systems on single-task editing, and outperforms four free-text editors on compound instructions for track removal, extraction and denoising on real (MoisesDB) and synthetic (Slakh) recordings. In a user study (16 listeners), ProCraft was rated highest on removal and denoising and on-par with a 16-times larger model on extraction and instrument transfer. Ablations further show that representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.
AI-generated analysis. Check claims against the paper.
Summary
ProCraft introduces a lightweight autoregressive codec language model capable of performing diverse and compound music editing tasks from free-form text instructions. The paper makes a significant contribution by addressing the challenge of preserving unedited content while applying multiple simultaneous edits, using novel architectural components like a persistent edit head and intermediate layer alignment, along with a synthetic data generation pipeline that enables training on compound instructions.
Methodology
The paper proposes ProCraft, a 0.5B parameter autoregressive codec language model for music editing. The methodology is sound, leveraging the InspireMusic backbone and WavTokenizer. The introduction of a "persistent edit head" with gated cross-attention to maintain instruction relevance over long sequences is a logical architectural choice for autoregressive models. The "On-the-fly Editing-pair Synthesis" is a clever data generation strategy using a tool-calling LLM to create compound instruction-audio pairs, addressing a significant data scarcity issue in this niche. The alignment of intermediate layers with MERT and pitch/instrument targets is a strong technical contribution to preserve unedited content.
Results & evaluation
The evaluation is comprehensive, covering seven operations across eight instruments. The use of both synthetic (Slakh) and real (MoisesDB) data for out-of-domain testing is rigorous. The metrics are well-chosen, combining general audio metrics (FAD, CLAP) with task-specific ones (SI-SDR, edit rate, preservation metrics). The user study, while small (16 listeners), provides necessary subjective validation. The model outperforms baselines on compound instructions, which is the core claim.
Reproducibility
The paper promises to release code and weights. The training procedure, including the two-pass training for the edit head, is described in detail. The data synthesis pipeline is explained, though the specific prompts and tool-calling logic for the LLM are likely in the appendix. The use of standard components (WavTokenizer, MERT) aids reproducibility.
Limitations
The model is limited to 30-second excerpts, which is short for full music production workflows. The user study is small. The model is trained primarily on synthetic data (Slakh) and tested on real data, which may limit generalization to complex, real-world mixes with non-standard instruments or production styles. The 0.5B parameter size, while efficient, may limit the complexity of creative edits compared to larger models.
Broader impact
This work contributes to the field of controllable audio generation by demonstrating that a single lightweight model can handle diverse and compound editing tasks. It bridges the gap between task-specific models and general-purpose text-guided editors. The data synthesis technique is applicable to other audio editing tasks. ProCraft introduces a lightweight autoregressive codec language model capable of performing diverse and compound music editing tasks from free-form text instructions. The paper makes a significant contribution by addressing the challenge of preserving unedited content while applying multiple simultaneous edits, using novel architectural components like a persistent edit head and intermediate layer alignment, along with a synthetic data generation pipeline that enables training on compound instructions.
Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for…
Abstract
Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for predicting human-annotated mean opinion scores (MOS), particularly for overall musical impression (OMI) and textual alignment (TA). However, CLAP is limited in capturing fine-grained musical attributes such as timbre and expressiveness \emph{etc}. To address this, we propose FUSEMOS, a dual-encoder fusion framework that integrates CLAP with MERT, a self-supervised model pretrained on large scale music audio, to better capture both high level semantic alignment and detailed musical characteristics for more accurate OMI and TA estimation. Furthermore, to overcome the limitations of conventional regression losses in measuring human preferred relative ranking of music segments, we design a novel ranking-aware composite loss function. Experiments conducted on the MusicEval benchmark demonstrate that our approach outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin.
AI-generated analysis. Check claims against the paper.
Summary
FUSEMOS proposes a dual-encoder fusion framework combining CLAP and MERT with a ranking-aware composite loss to improve the accuracy of automatic perceptual evaluation for text-to-music generation systems. The paper demonstrates significant improvements in rank correlation metrics over single-encoder baselines on the MusicEval benchmark, validating the complementary nature of semantic and structural audio representations for MOS prediction.
Methodology
The paper proposes FUSEMOS, a dual-encoder framework that fuses features from CLAP (for semantic alignment) and MERT (for musical structure) to predict MOS scores for Overall Musical Impression (OMI) and Textual Alignment (TA). The core architectural contribution is a "late fusion" strategy where independent MLP heads predict scores from each encoder, which are then combined via a linear layer. This is a standard ensemble technique rather than a deep feature-level fusion. The loss function combines a truncated regression loss (Huber-like) with a contrastive ranking loss. While the motivation to capture ordinal preferences is sound, the specific formulation of the ranking loss is a minor adaptation of existing contrastive learning objectives. The methodology is competent but relies heavily on existing pretrained models without significant architectural innovation in the fusion mechanism itself.
Results & evaluation
Experiments are conducted on the MusicEval benchmark, which is a relevant and recent dataset for TTM evaluation. The results show substantial improvements over the CLAP-only baseline, particularly in rank correlation metrics (SRCC, KTAU). The ablation studies are thorough, validating the necessity of both encoders, the choice of late fusion over early fusion/attention, and the components of the composite loss. The improvement in S-SRCC from 0.861 to 0.940 for TA is significant. However, the comparison is limited to the baseline provided by the dataset creators; comparisons with other recent MOS prediction methods (e.g., UTMOS variants adapted for music, or other dual-encoder approaches) are absent.
Reproducibility
The paper provides specific hyperparameters (learning rate, batch size, margins) and references specific pretrained checkpoints (HuggingFace links for CLAP and MERT). The use of a single H800 GPU and SGD optimizer is noted. However, the code repository is not provided in the text, and the specific implementation details of the "learnable query-based attention pooling" in the MERT branch are described briefly. The reliance on a specific, potentially proprietary or limited-access dataset (AISHELL_7A) may hinder full reproduction by the broader community if the dataset is not widely available.
Limitations
The primary limitation is the lack of generalization testing. The model is trained and tested on the MusicEval benchmark, which consists of clips from 31 specific models. It is unclear how well FUSEMOS generalizes to music generated by unseen models or different TTM architectures. Additionally, the "late fusion" approach is computationally redundant as it requires running two large encoders and multiple MLP heads, which could be a bottleneck for real-time evaluation. The paper does not discuss the computational cost or latency of the inference process.
Broader impact
This work contributes to the development of objective metrics for text-to-music generation, which is a rapidly growing field. By improving the correlation between automatic scores and human perception, it enables more efficient iteration and benchmarking of TTM systems. The integration of musical structure models (MERT) with semantic alignment models (CLAP) sets a precedent for multimodal evaluation frameworks. However, the impact is currently limited to the specific domain of TTM evaluation and does not extend to broader audio quality assessment tasks. FUSEMOS proposes a dual-encoder fusion framework combining CLAP and MERT with a ranking-aware composite loss to improve the accuracy of automatic perceptual evaluation for text-to-music generation systems. The paper demonstrates significant improvements in rank correlation metrics over single-encoder baselines on the MusicEval benchmark, validating the complementary nature of semantic and structural audio representations for MOS prediction.
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely…
Abstract
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
AI-generated analysis. Check claims against the paper.
Summary
LiveFoley introduces a streaming causal autoregressive diffusion architecture that achieves imperceptible latency and high-quality audio-visual alignment by decoupling frontend look-ahead from receptive field size via a Trailing Sliding Window and enhancing backend denoising with historical and future-warmup contexts. The technical contribution is significant for the field of real-time multimodal generation, offering a practical solution to the latency-quality trade-off that has hindered the adoption of high-fidelity V2A models in interactive environments.
Methodology
The paper proposes LiveFoley, a streaming video-to-audio generation system that addresses the latency-quality trade-off in real-time applications. The core contribution is the "Trailing Sliding Window" mechanism, which allows non-causal segment-level encoders (like Synchformer) to operate in a streaming context without retraining by anchoring the window tail to the current frame and using first-frame padding. This decouples look-ahead latency from the receptive field size. Additionally, the "Unified Varhop Framework" introduces a Synchronizer module to handle variable hop sizes, allowing the system to adapt to different latency budgets dynamically. For the audio backend, the authors modify the Local Diffusion Transformer (LocDiT) by introducing "History LocDiT" (using past context) and "Future-Warmup LocDiT" (using approximate future conditions) to improve generation quality compared to isolated patch denoising. The methodology is sound and logically addresses the identified bottlenecks in existing streaming V2A systems.
Results & evaluation
The paper includes a user study to quantify human perceptual thresholds for audio-visual delay, which is a valuable contribution to the field. Experiments demonstrate that LiveFoley matches offline models in temporal alignment while maintaining low latency. It claims to lead streaming systems in generation quality. The use of a single A800 GPU for real-time performance is a strong practical result. However, the text provided is truncated, so the full extent of quantitative comparisons (e.g., FAD, CLAP scores) against specific baselines like V-AURA or SoundReactor is not fully visible, though the abstract and intro suggest comprehensive testing.
Reproducibility
The paper provides architectural details and a project page. The use of standard components (MiniCPM, Synchformer, flow-matching) aids reproducibility. The specific implementation details of the Synchronizer and the training protocol for the variable hop sizes are described, but code availability is not explicitly confirmed in the provided text (only a project page URL is given).
Limitations
The system relies on a causal LM backbone which may still struggle with very long-range dependencies compared to global bidirectional models. The "Future-Warmup" mechanism introduces a small amount of additional latency (0.08s) which, while below perceptual thresholds, adds complexity. The evaluation is primarily focused on latency and alignment; subjective audio quality metrics (MOS) are likely included in the full paper but are not detailed in the truncated text.
Broader impact
This work has significant potential for immersive interactive applications such as AR/VR, video games, and live streaming, where real-time audio feedback is crucial. By providing a plug-and-play sound engine that works with arbitrary visual pipelines, it lowers the barrier for integrating high-quality synchronized audio into interactive media. LiveFoley introduces a streaming causal autoregressive diffusion architecture that achieves imperceptible latency and high-quality audio-visual alignment by decoupling frontend look-ahead from receptive field size via a Trailing Sliding Window and enhancing backend denoising with historical and future-warmup contexts. The technical contribution is significant for the field of real-time multimodal generation, offering a practical solution to the latency-quality trade-off that has hindered the adoption of high-fidelity V2A models in interactive environments.
Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from…
Abstract
Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.
AI-generated analysis. Check claims against the paper.
Summary
The paper proposes CRESiST, an end-to-end interleaved read-write framework for simultaneous speech translation that leverages an omni-modal LLM and an Actor-Critic self-enhancement pipeline to optimize streaming policies without rigid offline alignments. This work represents a strong technical contribution to the field of LLM-integrated audio processing, addressing key bottlenecks in latency and cache efficiency for infinite-length inference. While it does not reach the level of a paradigm-shifting landmark like the Transformer or BERT, it offers a significant architectural improvement over existing prefix-incremental methods, establishing new state-of-the-art Pareto frontiers for quality and latency. The integration of reinforcement learning (Actor-Critic) for policy distillation in a speech translation context is a novel and meaningful advance, likely to be cited by researchers working on streaming speech models and real-time translation systems.