Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio
Primary: University College London
All Institutions: University College London, King's College London
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
MFlowAudio introduces a novel latent audio generation framework that combines Flow Matching with a custom-designed TCMamba backbone. The core methodological innovation lies in the TCMamba block, which addresses the limitations of Mamba for 2D spectrograms by proposing a dual-scan mechanism: TimeMamba for long-range causal dependencies along the temporal axis (linear complexity) and ChannelMamba for cross-channel correlations like harmonic structures. This is a clever adaptation of SSMs to the audio domain, specifically designed for 2D latent representations. The formulation of Stateful Flow Matching (SFM) is another significant contribution, enabling chunk-wise training and streaming generation with O(1) caching complexity. This directly tackles the memory and computational bottlenecks of traditional diffusion/flow models for long-form audio. The theoretical analysis provided in the appendix, demonstrating the equivalence of chunked and one-time integration under certain causal assumptions, provides a strong foundation for SFM. Furthermore, the State-Decoupled Conditional Guidance (SDCG) mechanism is a novel approach to handle dynamic prompt switching, aiming to neutralize vector field collisions during on-the-fly semantic transitions. This mechanism, including the Orthogonal Score Projection, is well-motivated to balance prompt adherence and acoustic continuity. The overall architecture is clearly described, leveraging standard components like VAE for audio encoding and T5 for text encoding, integrated with the TCMamba blocks. The use of FiLM and cross-attention for conditioning is standard but well-integrated.
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.
Primary: Institute of Neuroinformatics, University of Zurich and ETH Zurich
All Institutions: Institute of Neuroinformatics, University of Zurich, ETH Zurich, Swiss Data Science Center, CVLab, EPFL
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
The paper proposes a rigorous methodology for constructing a physically verified benchmark for vocal attribution. The core innovation is the use of an on-body accelerometer-derived vibration signal to ground-truth caller identity, which is then withheld from the benchmark models to prevent shortcut learning. This is a sophisticated design choice that distinguishes the work from standard active speaker detection datasets that rely on perceptual annotation or close-talk references. The pipeline is detailed and auditable, involving synchronized multimodal data (5-channel audio, 3-camera video, 3D pose, radio telemetry) and a multi-stage verification process including manual review and geometric filtering. The evaluation protocol is particularly strong, defining three distinct regimes (session-disjoint, held-out-experiment, leave-one-individual-out) to explicitly separate in-domain recognition from transferable spatial reasoning.
The experimental evaluation is comprehensive, testing a wide array of baselines ranging from classical physics-based methods (SRP) to deep learning architectures (SELDnet++, EINV2, CST-former). The results clearly demonstrate the paper's central thesis: models that rely on familiar voice identity perform well in-domain but fail on unseen individuals, while spatially-aware models transfer better. The inclusion of modality intervention experiments (zeroing audio, shuffling geometry) provides strong causal evidence for what drives performance. The "on-body reference" ceiling confirms the consistency of the labels. The analysis of "close birds" and localization diagnostics adds depth to the evaluation.
Reproducibility is high. The authors release the full dataset, code for all evaluated methods, and fixed evaluation protocols. The paper provides extensive details on the data acquisition pipeline, camera calibration, and label construction in the appendices. The use of standard open-source tools (YOLOX, OC-SORT, Whisper) for preprocessing further aids reproducibility. The clear separation of the verification channel from the benchmark inputs ensures that future researchers can fairly evaluate new methods without access to the ground-truth sensor.
The primary limitation is the scale and diversity of the data. The dataset is limited to 10 zebra finches across 3 experiments in a single installation. This limits the generalizability of the findings to other species or larger, more dynamic groups. The radio telemetry features are limited due to carrier drift, which the authors acknowledge. The benchmark currently only covers single-vocalizer events, leaving multi-caller scenarios for the unevaluated V33DA++ release.
This work has significant impact on the fields of bioacoustics, multimodal learning, and behavioral analysis. By providing a physically verified benchmark, it enables the development of more robust and generalizable models for animal communication. The insights into shortcut learning and the importance of spatial reasoning are applicable to other multimodal attribution tasks, such as human active speaker detection. The release of the dataset and code will likely spur further research in this niche but important area. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
A speech deepfake detector estimates whether a recording is synthetic, but a forensic analyst also asks which system produced it, whether recordings share a source, and what evidence supports each conclusion. We present VeriSpoof, an audio language model that unifies five tasks with shared weights: deepfake detection, generator attribution (generator, acoustic model and vocoder), recording-condition analysis, synthesized-span localisation, and source verification. Each decision is supported by grounded rationales whose statements can be independently verified. Answers are scored by sentence likelihood, allowing detection and source verification to be thresholded without separate heads. Rationale statements can be checked against waveform measurements, span labels, or generator metadata. Counterfactual examples train decisions to follow the stated evidence. Across the 14 Speech-DF-Arena test sets, VeriSpoof-Plus achieves 2.14% pooled EER, ranking third of 28 systems and first among open-weight systems. VeriSpoof-Base, trained only on official training splits of eight corpora, achieves 4.43% EER, less than half that of the next open-weight detector. With the same weights, VeriSpoof-Plus verifies whether recordings from generators unseen in training share a source at 6.2% EER. Controlled studies show that the audio representation governs detection, the LLM governs recording comparison, and source supervision is necessary for neural-codec fakes. Statement-checkable rationales catch partial fakes missed by one-step answers (99.6% vs. 78.4% on PartialEdit) and determine decisions under edits unseen in training (99.5%). Training updates only 29.7M of 2.85B parameters, fitting a single 24 GB GPU with gradient checkpointing. We release both models, inference code, a demo, and VeriSpoof-Bench, the evaluation suite covering these questions.
Primary: The University of Melbourne
All Institutions: The University of Melbourne
VeriSpoof unifies multiple speech forensics tasks into a single audio language model with verifiable, grounded rationales. The paper makes a substantial contribution by demonstrating that LLMs can perform complex forensic reasoning (attribution, source verification) with high accuracy when equipped with checkable intermediate steps, outperforming specialized detectors on open-weight benchmarks and providing a new standard for explainable AI in audio security.
The paper proposes VeriSpoof, an audio language model (ALM) that unifies five distinct forensic tasks (deepfake detection, generator attribution, recording-condition analysis, synthesized-span localisation, and source verification) into a single architecture. The core methodological innovation is the "grounded rationale" framework, where the model generates explanations composed of fixed-field statements (e.g., SNR, bandwidth, span labels) that can be independently verified against waveform measurements or metadata. This addresses the "faithfulness" problem in LLM explanations by training with counterfactual examples that force the decision to depend on the stated evidence rather than just the audio input. The model uses a Qwen3-ASR encoder and MiniCPM5-2B LLM, employing parameter-efficient fine-tuning (LoRA/DoRA) on only 29.7M of 2.85B parameters. The scoring mechanism reads the likelihood of specific answer sentences, eliminating the need for separate classification heads for different tasks.
The experimental evaluation is extensive and rigorous. On the Speech-DF-Arena benchmark (14 test sets), VeriSpoof-Plus achieves a pooled EER of 2.14%, ranking third overall and first among open-weight systems. VeriSpoof-Base, trained on fewer data, achieves 4.43% EER, significantly outperforming other open-weight detectors. The paper includes controlled ablation studies to determine the contribution of the encoder vs. the LLM, showing that the audio representation governs detection while the LLM governs source comparison. It also demonstrates that source supervision is critical for detecting neural-codec fakes. The rationale mechanism is shown to catch partial fakes (99.6% vs 78.4% for one-step answers) and maintain decision faithfulness under unseen edit types (99.5%).
High. The authors release both model weights (Base and Plus), inference code, a demo, and a comprehensive evaluation suite (VeriSpoof-Bench) with 177,852 items. The training recipe, data composition, and hyperparameters are detailed. The claim that training fits on a single 24GB GPU with gradient checkpointing is supported by memory analysis, making the work accessible to a broader research community.
The model relies on specific base models (Qwen3-ASR, MiniCPM5-2B) which may not be the most optimal for all audio tasks. The "grounded rationale" approach requires careful construction of checkers for each statement type, which may not generalize automatically to new forensic questions without manual checker design. The performance on unseen languages (SpoofRadar) shows a calibration cost where pooled metrics hide per-set threshold issues. The paper is a preprint (arXiv) and has not yet undergone peer review at a major conference.
This work has significant implications for digital forensics and media integrity. By providing explainable, verifiable, and multi-functional deepfake detection, it moves the field beyond simple binary classification toward actionable forensic analysis. The ability to attribute generators and verify sources with grounded evidence could aid law enforcement and platform moderation. The release of open-weight models and benchmarks lowers the barrier to entry for developing forensic audio tools. VeriSpoof unifies multiple speech forensics tasks into a single audio language model with verifiable, grounded rationales. The paper makes a substantial contribution by demonstrating that LLMs can perform complex forensic reasoning (attribution, source verification) with high accuracy when equipped with checkable intermediate steps, outperforming specialized detectors on open-weight benchmarks and providing a new standard for explainable AI in audio security.
A speech deepfake detector estimates whether a recording is synthetic, but a forensic analyst also asks which system produced it, whether recordings share a source, and what evidence supports each conclusion. We present VeriSpoof, an audio language model that unifies five tasks with shared weights: deepfake detection, generator attribution (generator, acoustic model and vocoder), recording-condition analysis, synthesized-span localisation, and source verification. Each decision is supported by grounded rationales whose statements can be independently verified. Answers are scored by sentence likelihood, allowing detection and source verification to be thresholded without separate heads. Rationale statements can be checked against waveform measurements, span labels, or generator metadata. Counterfactual examples train decisions to follow the stated evidence. Across the 14 Speech-DF-Arena test sets, VeriSpoof-Plus achieves 2.14% pooled EER, ranking third of 28 systems and first among open-weight systems. VeriSpoof-Base, trained only on official training splits of eight corpora, achieves 4.43% EER, less than half that of the next open-weight detector. With the same weights, VeriSpoof-Plus verifies whether recordings from generators unseen in training share a source at 6.2% EER. Controlled studies show that the audio representation governs detection, the LLM governs recording comparison, and source supervision is necessary for neural-codec fakes. Statement-checkable rationales catch partial fakes missed by one-step answers (99.6% vs. 78.4% on PartialEdit) and determine decisions under edits unseen in training (99.5%). Training updates only 29.7M of 2.85B parameters, fitting a single 24 GB GPU with gradient checkpointing. We release both models, inference code, a demo, and VeriSpoof-Bench, the evaluation suite covering these questions.
Primary: The University of Melbourne
All Institutions: The University of Melbourne
VeriSpoof unifies multiple speech forensics tasks into a single audio language model with verifiable, grounded rationales. The paper makes a substantial contribution by demonstrating that LLMs can perform complex forensic reasoning (attribution, source verification) with high accuracy when equipped with checkable intermediate steps, outperforming specialized detectors on open-weight benchmarks and providing a new standard for explainable AI in audio security.
The paper proposes VeriSpoof, an audio language model (ALM) that unifies five distinct forensic tasks (deepfake detection, generator attribution, recording-condition analysis, synthesized-span localisation, and source verification) into a single architecture. The core methodological innovation is the "grounded rationale" framework, where the model generates explanations composed of fixed-field statements (e.g., SNR, bandwidth, span labels) that can be independently verified against waveform measurements or metadata. This addresses the "faithfulness" problem in LLM explanations by training with counterfactual examples that force the decision to depend on the stated evidence rather than just the audio input. The model uses a Qwen3-ASR encoder and MiniCPM5-2B LLM, employing parameter-efficient fine-tuning (LoRA/DoRA) on only 29.7M of 2.85B parameters. The scoring mechanism reads the likelihood of specific answer sentences, eliminating the need for separate classification heads for different tasks.
The experimental evaluation is extensive and rigorous. On the Speech-DF-Arena benchmark (14 test sets), VeriSpoof-Plus achieves a pooled EER of 2.14%, ranking third overall and first among open-weight systems. VeriSpoof-Base, trained on fewer data, achieves 4.43% EER, significantly outperforming other open-weight detectors. The paper includes controlled ablation studies to determine the contribution of the encoder vs. the LLM, showing that the audio representation governs detection while the LLM governs source comparison. It also demonstrates that source supervision is critical for detecting neural-codec fakes. The rationale mechanism is shown to catch partial fakes (99.6% vs 78.4% for one-step answers) and maintain decision faithfulness under unseen edit types (99.5%).
High. The authors release both model weights (Base and Plus), inference code, a demo, and a comprehensive evaluation suite (VeriSpoof-Bench) with 177,852 items. The training recipe, data composition, and hyperparameters are detailed. The claim that training fits on a single 24GB GPU with gradient checkpointing is supported by memory analysis, making the work accessible to a broader research community.
The model relies on specific base models (Qwen3-ASR, MiniCPM5-2B) which may not be the most optimal for all audio tasks. The "grounded rationale" approach requires careful construction of checkers for each statement type, which may not generalize automatically to new forensic questions without manual checker design. The performance on unseen languages (SpoofRadar) shows a calibration cost where pooled metrics hide per-set threshold issues. The paper is a preprint (arXiv) and has not yet undergone peer review at a major conference.
This work has significant implications for digital forensics and media integrity. By providing explainable, verifiable, and multi-functional deepfake detection, it moves the field beyond simple binary classification toward actionable forensic analysis. The ability to attribute generators and verify sources with grounded evidence could aid law enforcement and platform moderation. The release of open-weight models and benchmarks lowers the barrier to entry for developing forensic audio tools. VeriSpoof unifies multiple speech forensics tasks into a single audio language model with verifiable, grounded rationales. The paper makes a substantial contribution by demonstrating that LLMs can perform complex forensic reasoning (attribution, source verification) with high accuracy when equipped with checkable intermediate steps, outperforming specialized detectors on open-weight benchmarks and providing a new standard for explainable AI in audio security.
Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.
Primary: Northwestern University
All Institutions: Northwestern University, Kellogg School of Management
The paper introduces VIPBench, a comprehensive benchmark for evaluating the alignment between speech representation models and human perception of voice identity. It makes a significant contribution to the field by providing a large-scale dataset of human judgments on real, cloned, and morphed voices, along with a rigorous evaluation protocol that reveals the limitations of current speaker verification models in capturing human perceptual nuances.
The paper introduces VIPBench, a large-scale benchmark (124,876 judgments) designed to evaluate how well speech representation models align with human perception of voice identity, rather than just metadata-based speaker labels. The methodology is rigorous, involving a stratified sampling of 100 public figures across sociophonetic groups, genders, and ages. The stimuli include real recordings, state-of-the-art voice clones (Cartesia Sonic 3), and continuous morphs. The evaluation protocol defines four distinct tasks: continuous prediction of perceived sameness, Platt-calibrated verification, speaker-level representational similarity, and real-to-synthetic calibration transfer. The use of cross-validation over speakers and the inclusion of a noise ceiling based on inter-rater reliability demonstrates a sophisticated statistical approach to benchmarking.
The experiments evaluate 11 publicly available models, including supervised (x-vector, ECAPA-TDNN, Resemblyzer) and self-supervised (wav2vec 2.0, HuBERT, WavLM, XLS-R) models, as well as Whisper. The results reveal that while supervised models generally outperform self-supervised ones in aligning with human perception, none reach the noise ceiling. A key finding is the significant divergence between metadata labels and human perception for voice clones, where listeners often perceive clones as different speakers. The paper provides detailed ablations on layer selection for SSL models and robustness checks using unfamiliar trials and different TTS systems (CosyVoice 2). The inclusion of demographic fairness analysis is a strong addition, showing consistent performance across sociophonetic groups for most models.
The paper is highly reproducible. The dataset, code, and embeddings are released on Hugging Face. The authors provide detailed information on stimulus generation, including the specific TTS settings and random seeds used for morphing. The analysis notebook is provided, allowing users to reproduce the results without needing to re-extract embeddings. The clear documentation of the IRB protocol and data collection process further enhances trust and reproducibility.
The benchmark is limited to American English speakers and public figures, which may not generalize to other languages or private individuals. The reliance on a single commercial TTS system (Cartesia) for the main set of clones and morphs could introduce system-specific biases, although a small extension with CosyVoice 2 mitigates this partially. The lack of test-retest reliability for individual listeners is a minor limitation, though the high inter-rater reliability compensates for this.
This work has significant implications for the development of voice biometric systems, cybersecurity, and intellectual property protection. By highlighting the gap between metadata-based speaker verification and human perception, the paper motivates the training of speaker models directly on human perceptual judgments. This could lead to more robust and human-aligned voice authentication systems that are better equipped to handle voice cloning attacks. The benchmark also provides a valuable resource for researchers studying human voice perception and the alignment of machine representations with human cognition. The paper introduces VIPBench, a comprehensive benchmark for evaluating the alignment between speech representation models and human perception of voice identity. It makes a significant contribution to the field by providing a large-scale dataset of human judgments on real, cloned, and morphed voices, along with a rigorous evaluation protocol that reveals the limitations of current speaker verification models in capturing human perceptual nuances.
Dialectal Arabic speech recognition remains constrained by the scarcity of dialect-resolved transcribed speech despite the abundance of web audio. We introduce GigaSpeech 3, a corpus of 39,000 hours and 10 million weakly transcribed speech segments covering Algerian, Egyptian, Iraqi, Saudi, and Syrian Arabic. Starting from 60 hours of manual transcription per dialect, we train labeling models using speech-recognition-biased self-supervised learning and apply them to dialect-matched web audio. Data selection and four rounds of noisy-student refinement for Saudi and Syrian Arabic yield the final corpus. We validate its utility through dialect-specific and multi-dialect speech recognition training and cross-domain evaluation. On GigaSpeechBench, the final dialect-specific models outperform their labeling models across all five dialects and achieve a macro-average word error rate of 23.86%, a 16.6% relative reduction from the strongest third-party system covering all five dialects, while outperforming the best third-party result on three dialects. A multi-dialect model using self-predicted dialect identities achieves a macro-average word error rate of 25.51% while also supporting Modern Standard Arabic. Across eight conditions from three external corpora, a GigaSpeech 3-trained Zipformer achieves the lowest word error rate among six evaluated models in every condition, while adapting the Qwen3 speech recognizer on GigaSpeech 3 improves performance across all eight conditions. We release the corpus and recipes as a foundation for dialect-aware Arabic speech processing.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, University of Chinese Academy of Sciences, The Chinese University of Hong Kong, Shenzhen, Tsinghua University, Dalian University of Technology, Alibaba Token Foundry, Fudan University, SpeechColab, Seasalt.ai
The paper presents GigaSpeech 3, a large-scale weakly labeled corpus for five Arabic dialects, constructed via ASR-biased self-supervised learning and noisy-student refinement, demonstrating that such weak supervision enables compact models to outperform strong third-party systems and large pretrained models in dialect-specific and multi-dialect recognition tasks.
The paper proposes a data-centric approach to low-resource dialectal Arabic ASR. The core methodology involves constructing a large-scale weakly labeled corpus (GigaSpeech 3) using a "VietASR" pipeline. This pipeline leverages ASR-biased self-supervised learning (SSL) to train labeling models from small manual seeds (60 hours/dialect), which are then applied to dialect-matched web audio. The process includes data selection and iterative Noisy Student Training (NST) refinement for specific dialects (Saudi, Syrian). The novelty lies not in a new model architecture, but in the rigorous application of existing SSL and weak-labeling techniques to a specific, under-served linguistic domain (Arabic dialects) at a significant scale (39k hours). The validation strategy is sound: training independent downstream models (Zipformer) from scratch on the weak labels to prove the utility of the data beyond the labeling model's parameters.
The experimental evaluation is comprehensive and rigorous. The authors validate the corpus by showing that models trained on the weak labels outperform the labeling models themselves, confirming the data's utility. They compare against strong third-party baselines (including commercial APIs and large open-source models like Qwen3 and Whisper) on GigaSpeechBench, achieving state-of-the-art results for compact models. Cross-domain evaluation on MASC, Casablanca, and SADA further demonstrates generalizability. The inclusion of a multi-dialect model with automatic language identification (LID) and the analysis of routing errors adds depth to the evaluation. The results are consistent and well-supported by ablation studies (e.g., Labeled-only vs. SSL+FT vs. Weak-labels).
High. The authors release the code, dataset, and demo. The pipeline is described in detail, including the specific stages of the VietASR recipe and the NST refinement process. The use of standard architectures (Zipformer) and public benchmarks (GigaSpeechBench) enhances reproducibility. The separation of training and test channels at the source level is a strong methodological choice that is clearly documented.
The primary limitation is the reliance on web audio, which may contain noise, code-switching, or non-dialectal content that is difficult to fully filter. The country-level dialect labels are coarse and may not capture intra-country variation. The multi-dialect model's performance is affected by LID errors, particularly for Egyptian and Syrian dialects, as shown in the confusion matrix. The paper focuses on five specific dialects, leaving other Arabic varieties unaddressed.
This work has significant impact on the field of low-resource speech recognition. By providing a large, dialect-resolved corpus and a validated pipeline for weak labeling, it enables the development of more accurate and inclusive ASR systems for Arabic speakers. The methodology can be adapted to other low-resource languages or dialects. The release of the corpus and tools will likely accelerate research in dialectal Arabic processing and serve as a benchmark for future work. The paper presents GigaSpeech 3, a large-scale weakly labeled corpus for five Arabic dialects, constructed via ASR-biased self-supervised learning and noisy-student refinement, demonstrating that such weak supervision enables compact models to outperform strong third-party systems and large pretrained models in dialect-specific and multi-dialect recognition tasks.
Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.
Primary: Carnegie Mellon University
All Institutions: Queen Mary University of London, Carnegie Mellon University, Cornell University
The paper introduces PianoBench, a benchmark and dataset for evaluating musical perception and reasoning in language models, revealing a significant gap between audio perception and symbolic reasoning capabilities in current models. Through a rigorous data pipeline and comprehensive experiments, the authors demonstrate that while frontier models excel at reasoning over explicit note events, open-weight models struggle with audio perception, and that interleaved musical instruction can improve zero-shot reasoning without chain-of-thought.
The paper introduces a novel data pipeline to construct a large-scale (20k hours) dataset of interleaved musical commentary and symbolic transcripts from instructional piano videos. The methodology is robust, utilizing source separation, speech recognition, and MIDI transcription to create aligned multimodal data. The benchmark construction is rigorous, employing LLMs for question generation and distractor creation, followed by human verification to ensure quality. The proposed "MIDI-text" tokenization scheme is a clever contribution, allowing text-only LLMs to process symbolic music efficiently. The experimental design effectively isolates perception (audio) from reasoning (symbolic) by comparing performance across different input modalities.
The evaluation is comprehensive, testing a wide range of open-weight and proprietary models. The key findingโthat open-weight models struggle with audio perception while frontier models excel at symbolic reasoning with CoTโis well-supported by the data. The mid-training experiment on Qwen 3.5 9B Base provides strong evidence that interleaved instruction data improves zero-shot reasoning without CoT, outperforming separated data. The inclusion of a no-music control is crucial for validating the necessity of the musical input.
High. The authors release the dataset, benchmark, and code. The data pipeline is described in detail, including the specific models used for transcription and separation. The hyperparameters for mid-training are provided. The use of specific commercial APIs (GPT 5.6, Gemini 3.1) limits full reproducibility for those specific results, but the open-weight model results are reproducible.
The benchmark is limited to piano music, which may not generalize to other instruments or genres. The reliance on YouTube instructional videos may introduce biases in repertoire and teaching style. The "MIDI-text" representation, while efficient, is a lossy compression of the original audio and may not capture timbral or expressive nuances. The small size of the benchmark (250 questions) limits statistical power for fine-grained analysis.
This work highlights a critical gap in current audio-language models: the ability to reason about music when given symbolic representations versus perceiving it from audio. It provides a valuable resource (PianoTalk dataset) for training models on fine-grained musical understanding. The findings suggest that current open-weight models need better supervision for audio perception, while frontier models' reasoning capabilities can be leveraged to generate better training data. This could drive future research in multimodal music understanding and the development of more perceptually grounded audio models. The paper introduces PianoBench, a benchmark and dataset for evaluating musical perception and reasoning in language models, revealing a significant gap between audio perception and symbolic reasoning capabilities in current models. Through a rigorous data pipeline and comprehensive experiments, the authors demonstrate that while frontier models excel at reasoning over explicit note events, open-weight models struggle with audio perception, and that interleaved musical instruction can improve zero-shot reasoning without chain-of-thought.
Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely separate. We present Spatial Transfusion, a unified first-order ambisonics (FOA) framework that extends a pretrained spatial audio understanding model to generation and editing through flow matching over VAE latents. We formulate editing as a structured scene transformation over a taxonomy of atomic edits. Keeping the pretrained backbone frozen, we introduce modality-based LoRA routing and a curriculum that progressively adds generation, editing, and natural-language interaction. Experiments show that Spatial Transfusion largely preserves spatial understanding while outperforming full fine-tuning in generation and editing. The resulting model supports controllable FOA generation and editing through both structured and natural-language interfaces. Audio samples are available at https://microsoft.github.io/spatial-transfusion/
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, Microsoft Corporation
Spatial Transfusion introduces a unified framework for FOA audio understanding, generation, and editing by extending a pretrained spatial audio language model with flow matching and modality-based LoRA routing. The paper demonstrates that this approach preserves spatial understanding while enabling high-quality, controllable generation and editing, outperforming full fine-tuning baselines and establishing a new paradigm for spatial audio manipulation.
The paper proposes Spatial Transfusion, a unified framework for First-Order Ambisonics (FOA) audio understanding, generation, and editing. The core methodological contribution is the adaptation of a pretrained spatial audio language model (Sci-Phi) using a "Transfusion" paradigm, which combines autoregressive text modeling with flow-matching for continuous audio latents. A key technical innovation is the use of modality-based LoRA routing, where separate LoRA adapters are applied to audio and text tokens, allowing the frozen backbone to learn generative capabilities without catastrophic forgetting of understanding tasks. The authors define a structured taxonomy of atomic edits (semantic, acoustic, spatial, temporal) and employ a four-stage curriculum to progressively introduce generation and editing capabilities. The approach effectively leverages the structured nature of spatial scenes to enable precise control over generation and editing, distinguishing it from generic audio generation models.
The experimental setup is rigorous, utilizing a held-out set from the FOA-MEIR dataset with real room impulse responses. The evaluation metrics are comprehensive, covering spatial fidelity (FAD-PSELD, FOA covariance), semantic alignment (CLAP), and editing-specific metrics (EditFLAM). The results demonstrate that the proposed LoRA routing strategy outperforms full fine-tuning in generation and editing while preserving understanding capabilities. Ablation studies on routing strategies (modality-based vs. task-based vs. dense) provide strong evidence for the effectiveness of the chosen approach. The inclusion of natural language instruction tuning (Stage 4) further validates the flexibility of the framework.
The paper provides detailed descriptions of the architecture, training curriculum, and hyperparameters. However, specific details on the FOA-VAE implementation and the exact simulation pipeline parameters are referenced to prior work (Sci-Phi, FOA-VAE), which may limit immediate reproducibility without access to those codebases. The availability of a demo page with audio samples is a positive indicator, but a full code release is not explicitly mentioned in the text provided.
The model is trained primarily on simulated data, which may limit its generalization to real-world noisy environments, although the evaluation uses real RIRs. The framework is specific to FOA, and extension to higher-order ambisonics or binaural audio is not explored. The editing operations are limited to the defined atomic taxonomy, and complex, multi-step edits may not be supported. The reliance on a specific pretrained backbone (Phi-4-MM) may constrain the model's scalability.
This work bridges the gap between spatial audio understanding and generation, offering a unified interface for immersive audio creation. The structured editing capabilities have potential applications in virtual reality, gaming, and film production, where precise control over soundscapes is crucial. The framework's ability to handle both structured JSON and natural language inputs makes it accessible to both technical and non-technical users. Spatial Transfusion introduces a unified framework for FOA audio understanding, generation, and editing by extending a pretrained spatial audio language model with flow matching and modality-based LoRA routing. The paper demonstrates that this approach preserves spatial understanding while enabling high-quality, controllable generation and editing, outperforming full fine-tuning baselines and establishing a new paradigm for spatial audio manipulation.
Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks. We open-source the training code and model checkpoints at https://github.com/facebookresearch/gslm2.
Primary: Meta Superintelligence Labs
All Institutions: Meta Superintelligence Labs, Universite Paris Dauphine-PSL, AMI Labs, Doctolib, Ecole des Hautes Etudes en Sciences Sociales
GSLM2 introduces a multi-stream speech representation learning framework that significantly improves the data efficiency and linguistic competence of generative spoken language models. By jointly modeling discrete units from multiple encoder layers using a shared backbone and cross-stream generative heads, the method effectively captures complementary phonetic and semantic information, substantially narrowing the performance gap with text-based language models and outperforming existing speech-text systems on key linguistic benchmarks.
The paper proposes GSLM2, a framework addressing the data efficiency gap between spoken and text language models. The core methodological contribution is twofold: (1) SpidR 2.0, an encoder pretraining method that uses differentiable prediction heads to induce high-quality discrete units at multiple levels of abstraction (layers) simultaneously, replacing post-hoc k-means with joint optimization; and (2) a multi-stream SLM architecture that jointly models these unit streams using a shared temporal backbone, stream-specific predictors, and an autoregressive generative head to capture cross-stream dependencies. The use of entropy regularization to stabilize code distributions and the specific training objective combining single-stream, self-distillation, and multi-stream generative losses are technically sound and well-motivated.
The experiments are extensive, scaling up to 6M hours of data and 6.5B parameters. The paper demonstrates that GSLM2 consistently improves linguistic competence and scales more efficiently with data compared to single-stream baselines. It claims to shrink the gap with text-based LMs by about two orders of magnitude on key benchmarks (sWuggy, sBLIMP, tSC) and outperforms competitive speech-text systems. The comparison against both speech-only and speech-text baselines provides a strong evaluation of the textless approach.
The authors state that they open-source the training code and model checkpoints at the provided GitHub URL. The paper provides detailed mathematical formulations for the encoder pretraining and SLM training objectives, which aids in understanding the implementation. However, specific hyperparameters for the 6M hour runs may be complex to replicate without the full codebase, but the release of checkpoints is a significant positive factor.
The paper focuses primarily on English speech (Libri-Light/LibriSpeech). While it claims generalizability, the evaluation is limited to specific linguistic tasks (sWuggy, sBLIMP, tSC) which may not fully capture the breadth of spoken language understanding. The computational cost of training multi-stream models at scale is high, though the data efficiency gains are highlighted. The paper is an arXiv preprint, so peer review status is pending.
This work is significant for the field of speech AI as it provides a scalable, textless approach to learning linguistic structure. By narrowing the gap with text LMs, it enables more robust spoken dialogue systems that do not rely on transcriptions, which is crucial for low-resource languages or real-time applications. The open-sourcing of the code and checkpoints will likely accelerate research in multi-stream speech representation learning. GSLM2 introduces a multi-stream speech representation learning framework that significantly improves the data efficiency and linguistic competence of generative spoken language models. By jointly modeling discrete units from multiple encoder layers using a shared backbone and cross-stream generative heads, the method effectively captures complementary phonetic and semantic information, substantially narrowing the performance gap with text-based language models and outperforming existing speech-text systems on key linguistic benchmarks.
Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. Audio samples, code, datasets, and pretrained models are available at https://zhishengzheng.com/transdubber/.
Primary: The University of Texas at Austin
All Institutions: The University of Texas at Austin
The paper introduces a rigorous framework for timeline-preserving speech-to-speech translation, combining a scalable synthetic data pipeline with component-specific reinforcement learning to achieve state-of-the-art temporal alignment without compromising translation quality.
The paper proposes a novel formulation of "timeline-preserving" Speech-to-Speech Translation (S2ST), distinguishing it from prior work that focuses solely on total duration matching or speaker identity. The core methodological contribution is a data construction pipeline that creates 642K exactly duration-matched English-Chinese speech pairs by synthesizing target speech on "source-defined temporal scaffolds." This involves extracting speech/pause regions from source audio, translating segments under local duration budgets, and rendering them back onto the source timeline. The training strategy is a progressive curriculum: (1) pause-aware ASR/translation, (2) natural-pace speech generation, (3) timeline-preserving supervised fine-tuning, and (4) verifier-guided reinforcement learning (GSPO). The RL stage is particularly notable for its component-specific credit assignment, where translation quality rewards are applied to text tokens, while duration and pause alignment rewards are applied to speech-codec tokens, avoiding the dilution of signals that occurs in scalar reward aggregation.
The experiments are rigorous, utilizing a 4,353-utterance test set spanning read (CoVoST2) and spontaneous (GigaSpeech) speech. The system significantly outperforms strong baselines (UniSS, Seamless, Cascaded) in temporal metrics: increasing the proportion of translations within 2% of source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550. Crucially, this is achieved without sacrificing translation quality (COMET 0.788 vs 0.784 for UniSS) or speaker similarity. Ablation studies effectively isolate the contributions of pause markers, the natural-pace warm-up stage, and individual reward components, demonstrating that the RL stage specifically refines temporal precision while partially offsetting the quality cost of strict synchronization.
The paper provides high reproducibility. It details the exact data construction steps, including VAD thresholds, duration budget calculations, and synthesis parameters. The authors commit to releasing code, datasets, and pretrained models. The use of open-source components (Qwen3-ASR, VoxCPM2, Hy-MT2) for the pipeline further aids reproducibility, although the specific "source-canvas rendering" logic is a custom contribution that requires careful implementation.
The primary limitation is the reliance on synthetic data for supervision. While the pipeline is scalable, the quality of the synthesized "timeline-preserving" targets depends heavily on the underlying TTS model (VoxCPM2) and the translation model's ability to adhere to strict character budgets. The evaluation is limited to English-Chinese; generalization to other language pairs with different prosodic structures is not tested. Additionally, the "timeline preservation" constraint may not always be desirable in all dubbing contexts, as noted in the related work, though the paper frames this as an application-specific choice.
This work has significant implications for the dubbing industry and real-time speech translation applications where temporal synchronization with visual cues or other audio tracks is critical. By demonstrating that autoregressive models can learn complex temporal structures directly from aligned waveforms without explicit frame-level control at inference, it opens the door to more natural and synchronized multilingual audio generation. The component-specific RL approach could be adapted to other multimodal generation tasks requiring precise alignment between different modalities. The paper introduces a rigorous framework for timeline-preserving speech-to-speech translation, combining a scalable synthetic data pipeline with component-specific reinforcement learning to achieve state-of-the-art temporal alignment without compromising translation quality.
Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio
Primary: University College London
All Institutions: University College London, King's College London
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
MFlowAudio introduces a novel latent audio generation framework that combines Flow Matching with a custom-designed TCMamba backbone. The core methodological innovation lies in the TCMamba block, which addresses the limitations of Mamba for 2D spectrograms by proposing a dual-scan mechanism: TimeMamba for long-range causal dependencies along the temporal axis (linear complexity) and ChannelMamba for cross-channel correlations like harmonic structures. This is a clever adaptation of SSMs to the audio domain, specifically designed for 2D latent representations. The formulation of Stateful Flow Matching (SFM) is another significant contribution, enabling chunk-wise training and streaming generation with O(1) caching complexity. This directly tackles the memory and computational bottlenecks of traditional diffusion/flow models for long-form audio. The theoretical analysis provided in the appendix, demonstrating the equivalence of chunked and one-time integration under certain causal assumptions, provides a strong foundation for SFM. Furthermore, the State-Decoupled Conditional Guidance (SDCG) mechanism is a novel approach to handle dynamic prompt switching, aiming to neutralize vector field collisions during on-the-fly semantic transitions. This mechanism, including the Orthogonal Score Projection, is well-motivated to balance prompt adherence and acoustic continuity. The overall architecture is clearly described, leveraging standard components like VAE for audio encoding and T5 for text encoding, integrated with the TCMamba blocks. The use of FiLM and cross-attention for conditioning is standard but well-integrated.
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and present the SonoWorld4D framework for this task. Given a monocular video, our method generates a 360 static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time novel-view visual and spatial audio rendering. Evaluation on real-world dynamic scenes and a user study demonstrates SonoWorld4D generates 4D audio-visual scenes with perceptually realistic, physically grounded spatial audio consistent with scene geometry and visual content.
Primary: University of Maryland, College Park
All Institutions: University of Maryland, College Park
The paper introduces a novel task of Video-to-4DAVScene and a framework that effectively synchronizes visual dynamics with spatial audio using dual-space diffusion. The technical approach is rigorous, combining explicit 4D geometry with physically grounded acoustic rendering, and the introduction of a new dataset and evaluation protocol significantly advances the field of dynamic audio-visual scene generation.
The paper proposes SonoWorld4D, a framework for generating explorable 4D audio-visual scenes from a single monocular video. The methodology is technically sophisticated, relying on a decomposed 4D visual representation (static background Gaussians + dynamic foreground objects with explicit trajectories) and a novel "dual-space diffusion synchronization" mechanism. This synchronization couples camera-space audio generation with object-space audio generation via forward and inverse acoustic rendering operators during the diffusion sampling process. The use of flow-matching and the specific bidirectional consistency constraints (rendering object sounds to camera space and separating/inverting camera sounds to object space) represent a strong technical contribution to the field of spatial audio generation. The integration of VLM-derived prompts for acoustic context is also a relevant modern addition.
The authors introduce a new dataset (SonoScene4D) with 54 clips of dynamic scenarios featuring moving sound sources, recorded with synchronized FOA audio at novel listener poses. This is a valuable contribution to the field, as existing datasets often lack dynamic spatial audio ground truth. The evaluation includes objective metrics (DoA errors, CC, AUC, CLAP) and a user study with 50 participants. The results show significant improvements over baselines like Sonic4D and OmniAudio in both spatial accuracy and semantic consistency. The qualitative analysis of Doppler effects and novel listener adaptation is convincing.
The paper provides detailed descriptions of the pipeline, including the specific models used for depth estimation, segmentation, and Gaussian generation. However, the reliance on several proprietary or very recent models (e.g., HYWorld 2.0, Marble, SAM3, Kimi VLM) may limit immediate reproducibility for the community. The code is not explicitly linked in the provided text (only a demo page), which is a minor drawback for open science, though the demo page likely hosts assets.
The method assumes outdoor scenes with limited reverberation, omitting reflected paths and complex room acoustics, which limits its applicability to indoor environments. The pipeline is complex and relies on a chain of foundation models, which may introduce error propagation. The dataset size (54 clips) is relatively small for robust statistical validation, though it is a new task. The computational cost of the dual-space diffusion synchronization and 4D Gaussian rendering is likely high, though real-time rendering is claimed for the final scene.
This work bridges the gap between 4D visual generation and spatial audio synthesis, enabling the creation of immersive, interactive worlds from minimal input. It has potential applications in virtual reality, game asset generation, and film pre-visualization. The introduction of a benchmark for dynamic spatial audio generation is particularly impactful for the research community. The paper introduces a novel task of Video-to-4DAVScene and a framework that effectively synchronizes visual dynamics with spatial audio using dual-space diffusion. The technical approach is rigorous, combining explicit 4D geometry with physically grounded acoustic rendering, and the introduction of a new dataset and evaluation protocol significantly advances the field of dynamic audio-visual scene generation.
A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and image input modalities. Built on the post-trained Gemma 4 26B-A4B checkpoint, Apriel Voice directly processes speech and generates both responses and tool calls without requiring a separate dialogue model. We introduce a two-stage training procedure to adapt the model to speech. The first stage establishes the speech interface, while the second jointly adapts the projector and backbone through supervised fine-tuning, text replay, and source-dependent cross-entropy and forward Kullback--Leibler divergence supervision. Automatically constructed speech-conditioned dialogues provide conversational training data, while local teacher serving, overlapping teacher and student computation, and fused full-vocabulary losses enable efficient online distillation. In thinking mode, Apriel Voice achieves a score of 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and a task success rate of 65.51% on $ฯ$-Voice, with a mean latency of 1.91 seconds. It also achieves 51.27% English EVA accuracy and 74.13% experience pass@1. Finally, we deploy Apriel Voice 27B Preview on a custom serving platform to support interactive voice interactions.
Primary: ServiceNow
All Institutions: ServiceNow
Apriel Voice 27B Preview is an open speech-language model that unifies speech understanding, reasoning, and tool use within a shared backbone. The paper presents a rigorous two-stage adaptation recipe combining projector training with joint fine-tuning using source-dependent CE/FKL supervision, demonstrating strong performance in agentic speech tasks while retaining text and image capabilities.
The paper proposes a two-stage training procedure to adapt a text-and-image LLM (Gemma 4 26B-A4B) into a speech-language model. Stage 1 freezes the backbone and trains a speech projector (Whisper encoder + MLP) on transcription, translation, and continuation tasks. Stage 2 jointly fine-tunes the projector and backbone using a mixture of speech tasks, synthetic speech-conditioned dialogues, tool interactions, and text replay. A key technical contribution is the source-dependent loss function combining Cross-Entropy (CE) for task targets and Forward Kullback-Leibler (FKL) divergence from a frozen teacher model to preserve original capabilities. The system also details an efficient training infrastructure using local teacher serving, overlapping computation, and fused full-vocabulary loss kernels. While the individual components (projector, distillation, SFT) are established, the specific combination for agentic speech tasks and the engineering optimizations for online distillation are well-executed.
The evaluation is comprehensive, covering foundational speech metrics (WER, BLEU), reasoning (Big Bench Audio, GSM8K), instruction following (IFEval), and agentic task completion (EVA, ฯ-Voice, EnterpriseOps-Gym). The model demonstrates strong performance, particularly in reasoning (98.90 on BBA) and tool use (81.13 on BFCL), outperforming other open-source 2-stage models. However, the reliance on synthetic data for conversation training and the lack of ablation studies isolating the impact of specific training components (e.g., FKL vs. CE, specific data mixes) limit the depth of the empirical analysis. The comparison with proprietary systems is competitive but relies on different evaluation harnesses and simulators, making direct comparison difficult.
The paper provides detailed architectural specifications (projector dimensions, parameter counts) and training hyperparameters (learning rates, batch sizes, GPU configuration). The release of an interactive demo and the use of open-source base models (Gemma, Whisper) enhance reproducibility. However, the synthetic data generation pipeline and specific internal datasets are not fully released, and the "thinking mode" implementation details are not fully specified, which may hinder exact reproduction of the reported reasoning scores.
The model is primarily optimized for English, with limited multilingual speech understanding. The evaluation lacks paired significance tests against the base model, making it hard to quantify the exact cost of adaptation on text/image capabilities. The synthetic data pipeline may propagate errors from the TTS or LLM generators. The paper acknowledges that frozen visual components do not guarantee unchanged image reasoning, yet the drop in some vision benchmarks suggests potential degradation.
This work contributes to the development of open-source speech-language models capable of agentic tasks, which is a critical area for enterprise and consumer applications. By demonstrating that a single backbone can handle speech, text, images, and tool calls, it supports the trend toward unified multimodal agents. The efficient training techniques for online distillation are valuable for the broader community adapting large models to new modalities. Apriel Voice 27B Preview is an open speech-language model that unifies speech understanding, reasoning, and tool use within a shared backbone. The paper presents a rigorous two-stage adaptation recipe combining projector training with joint fine-tuning using source-dependent CE/FKL supervision, demonstrating strong performance in agentic speech tasks while retaining text and image capabilities.
Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.
Primary: Institute of Neuroinformatics, University of Zurich and ETH Zurich
All Institutions: Institute of Neuroinformatics, University of Zurich, ETH Zurich, Swiss Data Science Center, CVLab, EPFL
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
The paper proposes a rigorous methodology for constructing a physically verified benchmark for vocal attribution. The core innovation is the use of an on-body accelerometer-derived vibration signal to ground-truth caller identity, which is then withheld from the benchmark models to prevent shortcut learning. This is a sophisticated design choice that distinguishes the work from standard active speaker detection datasets that rely on perceptual annotation or close-talk references. The pipeline is detailed and auditable, involving synchronized multimodal data (5-channel audio, 3-camera video, 3D pose, radio telemetry) and a multi-stage verification process including manual review and geometric filtering. The evaluation protocol is particularly strong, defining three distinct regimes (session-disjoint, held-out-experiment, leave-one-individual-out) to explicitly separate in-domain recognition from transferable spatial reasoning.
The experimental evaluation is comprehensive, testing a wide array of baselines ranging from classical physics-based methods (SRP) to deep learning architectures (SELDnet++, EINV2, CST-former). The results clearly demonstrate the paper's central thesis: models that rely on familiar voice identity perform well in-domain but fail on unseen individuals, while spatially-aware models transfer better. The inclusion of modality intervention experiments (zeroing audio, shuffling geometry) provides strong causal evidence for what drives performance. The "on-body reference" ceiling confirms the consistency of the labels. The analysis of "close birds" and localization diagnostics adds depth to the evaluation.
Reproducibility is high. The authors release the full dataset, code for all evaluated methods, and fixed evaluation protocols. The paper provides extensive details on the data acquisition pipeline, camera calibration, and label construction in the appendices. The use of standard open-source tools (YOLOX, OC-SORT, Whisper) for preprocessing further aids reproducibility. The clear separation of the verification channel from the benchmark inputs ensures that future researchers can fairly evaluate new methods without access to the ground-truth sensor.
The primary limitation is the scale and diversity of the data. The dataset is limited to 10 zebra finches across 3 experiments in a single installation. This limits the generalizability of the findings to other species or larger, more dynamic groups. The radio telemetry features are limited due to carrier drift, which the authors acknowledge. The benchmark currently only covers single-vocalizer events, leaving multi-caller scenarios for the unevaluated V33DA++ release.
This work has significant impact on the fields of bioacoustics, multimodal learning, and behavioral analysis. By providing a physically verified benchmark, it enables the development of more robust and generalizable models for animal communication. The insights into shortcut learning and the importance of spatial reasoning are applicable to other multimodal attribution tasks, such as human active speaker detection. The release of the dataset and code will likely spur further research in this niche but important area. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.