Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.
Primary: Brown University
All Institutions: Brown University, MIT, Yale University, The University of Hong Kong
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Sonic-AMG introduces a novel framework for streaming human motion generation from spatial audio, addressing key limitations of prior offline methods. The core methodology is built upon a causal transformer with a flow head, operating on continuous motion latents from a pretrained VAE. A significant methodological contribution is the separation of acoustic content ("what") and body-relative source geometry ("where") into distinct conditioning tokens. This design choice, supported by ablations, allows the model to better disentangle and utilize these modalities. The 11-dimensional spatial descriptor, inspired by perceptual research, is a thoughtful design for capturing relevant spatial cues like proximity, relative motion, and looming. The paper effectively tackles two critical training-inference mismatches inherent in autoregressive streaming generation. First, the "history mismatch" (teacher-forced training vs. generated histories at inference) is addressed by "rehearsal training." This innovative approach combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. This hybrid strategy allows the model to learn stable long-range dependencies while also gaining robustness to accumulated errors from its own predictions. Second, the "spatial conditioning mismatch" (ground-truth SSL vs. generated root poses) is resolved by "spatial feedback." This mechanism recomputes body-relative source locations from generated root poses during both rehearsal training and inference, ensuring consistency between the generated motion and the spatial conditions. The integration of these components within a causal transformer and flow-matching framework is technically sound and well-justified. The use of classifier-free guidance during inference further enhances generation quality.
The experimental evaluation is comprehensive and well-structured, primarily using the SAM dataset. The paper establishes state-of-the-art performance for Sonic-AMG on FID and R-precision metrics, demonstrating both distributional fidelity and conditioning alignment. A crucial aspect of the evaluation is the distinction between "GT SSL" (ground-truth spatial conditioning, for fair comparison with baselines) and "Online SSL" (spatial feedback from generated motion, reflecting real-world streaming). Sonic-AMG maintains strong performance under the more challenging Online protocol, which is a significant achievement. The ablation studies are particularly strong, providing clear evidence for the efficacy of the proposed components: 1. **Rehearsal training vs. two-forward training:** Rehearsal training consistently outperforms various two-forward schedules, validating the benefit of recursive history construction. 2. **Role of teacher-forced supervision:** Demonstrates that combining self-rollouts with full-sequence supervision is crucial for fidelity and alignment. 3. **Effect of spatial feedback:** Shows that training with spatial feedback dramatically improves performance under the Online protocol, confirming its necessity for consistent spatial conditioning. 4. **Separate spatial tokens:** Ablation against a "Fused" approach confirms the advantage of distinct acoustic and spatial tokens. 5. **Learned acoustic features:** Shows that simpler, learned log-mel features outperform MOSPA's high-dimensional audio descriptors, supporting the hypothesis that separating "what" from "where" helps the acoustic branch focus on content. Beyond quantitative metrics, the paper provides compelling qualitative results, including examples of recognizable and spatially appropriate reactions, and the ability to generate coherent motion across complete audio clips and successive sound events. The demonstration of real-time streaming efficiency, with significantly reduced initial response latency compared to MOSPA, is a strong practical contribution. The inclusion of physical simulation retargeting to a Unitree G1 robot further showcases the robustness and applicability of the generated motions. The "No-SSL" variant, which derives spatial tokens directly from binaural audio, is an excellent exploration of alternative inference scenarios, demonstrating the model's flexibility and potential for deployment without explicit source location inputs.
The paper includes a dedicated "Reproducibility statement" and detailed appendices covering implementation details, training protocols, and runtime measurements. This includes specifics on architecture (transformer layers, heads, width, flow head details), optimization (AdamW, learning rate, batch size), training hardware and time, motion autoencoder pretraining, acoustic and spatial encoding, guidance, and training sequence construction for both teacher-forced and rehearsal training. The authors commit to releasing code and models upon acceptance, which, combined with the extensive details provided, suggests high reproducibility.
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Primary: FAIR, Meta Superintelligence Labs
All Institutions: The University of Texas at Austin, FAIR, Meta Superintelligence Labs
The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
The paper proposes RUBATO, a multistream positional encoding mechanism that assigns independent "clocks" to different data streams (real-time audio vs. internal thinking) in full-duplex speech models. The core innovation is decoupling the positional indices of asynchronous streams to prevent temporal distortion when the decoding rate of the thinking stream varies. The authors derive three variants: TM-RoPE (subspace partitioning), Key-Clock (conditional rotation based on key token clock), and Full-Clock (rotating all dimensions by the key's clock). The methodology is mathematically sound, leveraging the properties of Rotary Position Embeddings (RoPE) to allow zero-shot interpolation of inference speeds. The implementation strategy, which expands hidden dimensions during training to fit standard FlashAttention kernels and uses separate KV caches during inference, is a clever engineering solution that avoids custom Triton kernels, enhancing practicality.
The experiments are rigorous, evaluating the model on Spoken GSM8K and BIG-Bench Audio, including a novel "Overlapped" scenario where the model must handle interruptions while reasoning. The results demonstrate that RUBATO-Full maintains high accuracy (48-51%) across varying decoding budgets, whereas baselines collapse to near-zero accuracy at interpolated speeds. The paper provides strong evidence for the utility of the method in reducing time-to-answer by leveraging faster hardware without retraining. The inclusion of UTMOS and Speaker Similarity metrics for filler audio quality adds depth to the evaluation, confirming that the architectural changes do not degrade acoustic quality.
The paper provides a detailed reproducibility statement, including closed-form equations for the attention mechanisms and clear descriptions of the training data synthesis process. The use of standard backbones (Gemma-4) and tokenizers (NanoCodec) aids reproducibility. However, the specific hyperparameters for the training mixture and the exact construction of the synthetic datasets are deferred to the appendix, which is not fully visible in the provided text, though the main text describes the logic clearly.
The primary limitation is the reliance on synthetic data for training, as no suitable full-duplex spoken reasoning corpora exist. The evaluation is limited to mathematical and logical reasoning tasks (GSM8K, BIG-Bench), which may not generalize to all conversational domains. Additionally, the method assumes a specific interleaving strategy (SCoT), and its applicability to other full-duplex architectures (e.g., concatenation-based) is not explored. The "Full-Clock" variant, while best performing, requires duplicating the head dimension, which increases memory usage.
This work has significant implications for the deployment of real-time AI agents. By decoupling reasoning speed from real-time constraints, it enables more efficient use of heterogeneous hardware. The concept of independent clocks for asynchronous streams is a generalizable architectural pattern that could apply to other multimodal or multi-agent systems requiring variable-rate processing. It addresses a critical bottleneck in making LLMs truly interactive and responsive in speech-based interfaces. The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
Primary: Meta Superintelligence Labs
All Institutions: Meta Superintelligence Labs, University of Southern California
The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
The paper proposes ATTEST, a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) specifically tailored for long-audio temporal grounding. The core novelty lies in the design of three distinct reward signals: (1) a Counterfactual Audio-Grounding (CAG) reward that penalizes predictions surviving the removal of acoustic evidence (via silence or corruption), effectively acting as a hallucination penalty; (2) an Attention Density Grounding reward that encourages the model to attend to the specific audio tokens corresponding to the time spans it claims in its output; and (3) a token-wise fine-grained advantage mechanism that assigns credit at the segment level rather than the sequence level, which is crucial for long, dense captions. This approach avoids modifying the base model's architecture or vocabulary, making it a plug-and-play improvement for existing Large Audio Language Models (LALMs). The methodology is sound, addressing the specific failure modes of LALMs on long audio (hallucination and temporal drift) with targeted RL interventions.
The authors introduce EgoSCAPE, a new benchmark derived from Ego4D with 700 manually verified audio clips up to 5 minutes long, featuring dense captions, sound sources, and temporal grounding queries. This is a significant contribution as existing benchmarks often lack dense annotations for long-form audio. Experiments show that ATTEST outperforms strong baselines (including Gemini 3.1 Pro, GPT-4o Audio, and specialized models like TimeAudio and AudioFlamingo 3) on both the new EgoSCAPE benchmark and existing datasets (Clotho-Moment, CASTELLA). The ablation studies convincingly demonstrate the individual contribution of each reward component, particularly the CAG reward in reducing hallucinations. The results on general audio understanding benchmarks (MMAR, LongAudioBench) confirm that the method does not cause catastrophic forgetting, which is a common issue with temporal grounding fine-tuning.
The paper provides a detailed description of the benchmark creation pipeline, including the use of BEATs for initial detection and LLMs for verification, followed by human verification. The RL training details (LoRA, GRPO parameters, reward weights) are specified. However, the code and the EgoSCAPE benchmark itself are not explicitly linked in the provided text (though a reproducibility statement is made). The reliance on specific proprietary models (Gemini, GPT) for baselines limits full reproducibility of those comparisons, but the open-source base models (Qwen, AudioFlamingo) allow for partial reproduction.
The benchmark is limited to 5-minute clips, which is still shorter than some real-world long-form audio scenarios (e.g., hours-long podcasts). The method relies on the availability of attention weights, which may not be accessible in all proprietary or optimized inference engines. The counterfactual reward requires generating a reference model output on corrupted audio, adding computational overhead during training. The paper acknowledges that the method is currently evaluated on audio-only tasks and does not extend to video or omni-modal temporal grounding in the main experiments.
This work addresses a critical bottleneck in deploying LALMs for real-world applications where audio is long and unstructured. By providing a rigorous benchmark (EgoSCAPE) and a training framework (ATTEST) that reduces hallucinations without architectural changes, it enables more reliable temporal reasoning in audio AI. This has implications for accessibility tools, video indexing, and autonomous agents that need to understand and locate specific events in long audio streams. The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.
Primary: Unknown (Likely Alibaba/Tongyi Lab based on Qwen3/MOSS dependencies)
All Institutions: Unknown
Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
The paper proposes Soloist, a generative model for blind voice separation that frames the task as autoregressive generation of audio-codec sequences. The core methodological contribution is a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT) on synthetic mixtures with isolated targets and, crucially, on real recordings with only speaker timeline annotations (who speaks when), which helps the model learn count inference and activity patterns without waveform targets; and (2) Reinforcement Learning (RL) on real recordings using a "model-judge" reward. The RL component is particularly novel, employing a multi-step verification process where an audio-language model (Qwen3-Omni) describes candidates independently, and a text model (Qwen3.6) ranks them based on these descriptions, with a consistency check (Kendall's tau) under shuffled/reversed orders to filter out order-biased or unstable feedback. This addresses the lack of ground-truth isolated sources in real-world data. The architecture leverages a Qwen3-4B backbone with a local transformer for codec code prediction, integrating continuous and discrete audio features.
The evaluation is comprehensive, covering overlapping speech (LibriMix, WildTV), noisy environments (ViW), and singing vocals (SongYT, MIR-1K, MUSDB18-HQ). The paper reports strong results in PI-WER and DNSMOS, outperforming traditional separators (SepFormer, MossFormer) and query-based extractors. The inclusion of a full-duplex conversation pipeline demonstrates practical utility, showing how the model can be integrated into a larger system for continuous speaker track assembly. The judge validation section provides evidence that the RL reward signal correlates with objective quality metrics (PI-WER) on known-source mixtures, which is a critical sanity check for the RL approach.
The paper provides detailed descriptions of the data construction, training objectives, and RL hyperparameters. However, the reliance on specific, potentially proprietary or large-scale models (Qwen3-Omni, MOSS-Audio-Tokenizer, MOSS-Transcribe-Diarize) and the complexity of the judge pipeline make exact reproduction difficult without access to these specific components and the in-house datasets (e.g., the "in-house song recordings" and specific Pixabay subsets). The code is not explicitly linked in the provided text, though the use of open-source base models (Qwen3) suggests a path to partial reproduction.
The method is computationally expensive due to the autoregressive generation of codec codes and the multi-step RL judge process. The "blind" separation relies on the model's ability to infer the correct number of speakers, which can fail in highly overlapping or noisy scenarios. The RL reward, while robust to order bias, is still based on an LLM's judgment of audio descriptions, which may not perfectly align with human perceptual quality or subtle acoustic artifacts. The singing separation uses teacher-generated pseudo-labels, which limits the fidelity to the true studio vocal.
This work bridges the gap between traditional signal processing for speech separation and modern generative AI techniques. By using LLMs as judges for RL in audio tasks, it offers a template for training audio models on unpaired, real-world data. The application to full-duplex conversation processing highlights its potential for real-time or near-real-time conversational AI systems that require robust handling of overlapping speech and noise. Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.
Primary: Fish Audio
All Institutions: Fish Audio
Fish Audio Drama 3 introduces a 35B MoE speech model with a novel codebook-MTP decoder and preference-based RL, achieving state-of-the-art results in multilingual cloning and expressive performance while improving inference efficiency through architectural and engineering optimizations.
The paper introduces Fish Audio Drama 3, a 35B-parameter Mixture-of-Experts (MoE) speech generation model. The core architectural innovation is the replacement of the previous "Fast AR" module with a "Codebook Multi-Token-Prediction (MTP) Decoder." This decoder uses dedicated layers for each residual codebook that attend causally over the entire sequence, allowing fine acoustic details from reference audio to directly influence codebook predictions, rather than being compressed into a single backbone state. The backbone utilizes a hybrid of Gated DeltaNet (linear attention) and full attention layers to handle long contexts efficiently. A significant methodological contribution is the training pipeline, which employs a single learned reward model for Reinforcement Learning (RL) instead of rule-based rewards (like WER or speaker similarity). This addresses the conflict where expressive performance (whispering, laughing) often penalizes standard intelligibility/similarity metrics. The RL uses Group Relative Policy Optimization (GRPO) with large batches and no KL penalty, optimizing a preference model trained on human judgments.
The evaluation is comprehensive, covering multilingual cloning (MiniMax Multilingual Testset), editing (Ming-Freeform-Audio-Edit), and revision (Step-Audio-Edit). Drama 3 demonstrates state-of-the-art or competitive performance, particularly in multilingual cloning where it outperforms OmniVoice and FireRedTTS3-Base in multiple languages. The paper provides a strong ablation on the reward model, showing that the preference-based reward correctly prioritizes styled speech (e.g., whispering) over neutral speech, whereas rule-based rewards would penalize it. Inference efficiency is also rigorously evaluated, showing that despite having 7x more parameters than the previous model (S2), the new architecture and FP8 quantization allow for faster decoding (271 fps vs 177 fps on H200).
High. The model weights are released on Hugging Face and GitHub. The paper provides detailed descriptions of the architecture, training stages, and inference engine optimizations. While the specific data curation details are proprietary, the architectural and algorithmic details are sufficient for reproduction of the model structure. The use of open-source benchmarks and public baselines enhances reproducibility of the evaluation.
The model is large (35B parameters), which may limit deployment on edge devices despite the MoE efficiency. The paper relies on a proprietary reward model for RL, which is not released, making it difficult to fully reproduce the RL alignment process. The evaluation of "directed performance" is somewhat subjective and relies on the specific benchmarks provided, which may not cover all nuances of human direction.
This work significantly advances the field of expressive speech synthesis by unifying cloning, direction, editing, and multi-speaker scenes in a single model. The shift from rule-based to preference-based RL rewards is a crucial step towards more natural and controllable audio generation. The open-source release of such a large, high-performing model lowers the barrier for research and application in interactive characters, dubbing, and audiobooks. Fish Audio Drama 3 introduces a 35B MoE speech model with a novel codebook-MTP decoder and preference-based RL, achieving state-of-the-art results in multilingual cloning and expressive performance while improving inference efficiency through architectural and engineering optimizations.
Deep learning models of speech seldom represent the source-filter process by which speech is produced, relying instead on latent features with no physical correspondence to articulation or phonation. We propose SoFilNet, a self-supervised, differentiable Source-Filter Network that estimates the parameters of speech production directly from the raw waveform: a source, specified by fundamental frequency (F0), voicing and the aperiodicity of five frequency bands, and a filter, specified by ordered formant poles, additional poles that shape the rest of the spectral envelope, and antiresonance zeros. The model is trained only to reconstruct the input speech; no formant or pitch annotation is used. Since per-bin spectral losses favor blurred spectra wherever the model is uncertain, we train F0 and voicing against autocorrelation pseudo-labels refined with the model's own filter, aperiodicity against the periodic fraction of each band, and formant sharpness against spectral contrast. On VTR-FORMANTS, the estimated formants attain the lowest mean error among the reported methods, including supervised trackers, and on the unseen PTDB-TUG corpus the F0 frame error matches the best classical tracker and is lower than that of all learned trackers. The parameters are also consistent with speech production theory: aperiodicity rises with frequency and orders phone classes by the turbulence of their excitation, and the zeros deepen on nasal consonants, the lower one rising with place of articulation.
Primary: Indian Institute of Technology Hyderabad
All Institutions: Indian Institute of Technology Hyderabad
The paper presents a novel self-supervised differentiable source-filter model that accurately estimates speech production parameters from raw waveforms. By addressing the limitations of per-bin spectral losses with targeted auxiliary losses and a multi-resolution architecture, SoFilNet achieves state-of-the-art formant and pitch tracking accuracy while providing physically interpretable parameters consistent with speech production theory.
The paper proposes SoFilNet, a self-supervised differentiable source-filter model that estimates speech production parameters (F0, voicing, band aperiodicity, formants, zeros) directly from raw waveforms. The methodology is technically sophisticated, addressing the "blur" problem of per-bin spectral losses by introducing specific auxiliary losses: autocorrelation-based pseudo-labels for F0/voicing (refined via inverse filtering), periodic fraction loss for aperiodicity, and spectral contrast loss for formant sharpness. The use of a multi-resolution encoder with separate gates for source and filter parameters is a well-motivated architectural choice that aligns with the different temporal scales of glottal and vocal tract dynamics. The stick-breaking recursion for formant ordering is a clever constraint to ensure physical consistency without post-hoc tracking.
The evaluation is rigorous and comprehensive. The model is tested on VTR-FORMANTS for formant tracking, achieving the lowest mean error among reported methods, including supervised baselines. Pitch tracking is evaluated on the unseen PTDB-TUG corpus, where SoFilNet matches the best classical tracker (RAPT) and outperforms learned trackers. The interpretability analysis provides strong evidence that the learned parameters align with speech production theory (e.g., aperiodicity rising with frequency, zeros deepening on nasals). The ablation studies effectively demonstrate the contribution of zeros, the multi-resolution encoder, and the specific loss functions. The reconstruction quality (PESQ/STOI) from parameters alone further validates the information content of the estimated features.
The paper provides detailed descriptions of the architecture, loss functions, and training procedure. Hyperparameters such as learning rate, batch size, and number of epochs are specified. The use of standard datasets (TIMIT, VTR-FORMANTS, PTDB-TUG) and well-known baselines facilitates reproduction. However, the specific implementation details of the differentiable synthesizer and the exact code for the pseudo-label refinement process are not provided in the text, though a demo link is available.
The model is trained on TIMIT, which is a relatively small and dated dataset, potentially limiting generalization to diverse accents or noisy environments. The evaluation of formants is limited to F1-F3 (F4 excluded due to lack of hand-correction in the benchmark), which may not capture the full vocal tract response. The pitch evaluation, while strong, is on a single corpus (PTDB-TUG), and the model's performance on highly noisy or pathological speech is not assessed. The computational cost of the multi-resolution encoder and the iterative pseudo-label refinement may be higher than simpler baselines.
This work bridges the gap between data-driven deep learning models and physical speech production theory. By providing interpretable parameters that correspond to articulatory and phonatory features, SoFilNet can be used for speech analysis, synthesis, and modification tasks where physical interpretability is crucial. The self-supervised approach reduces the need for expensive expert annotations, making it scalable to larger datasets. The insights into the behavior of aperiodicity and zeros could inform future speech coding and synthesis models. The paper presents a novel self-supervised differentiable source-filter model that accurately estimates speech production parameters from raw waveforms. By addressing the limitations of per-bin spectral losses with targeted auxiliary losses and a multi-resolution architecture, SoFilNet achieves state-of-the-art formant and pitch tracking accuracy while providing physically interpretable parameters consistent with speech production theory.
Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
Primary: Meta Superintelligence Labs
All Institutions: Meta Superintelligence Labs, University of Southern California
The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
The paper proposes ATTEST, a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) specifically tailored for long-audio temporal grounding. The core novelty lies in the design of three distinct reward signals: (1) a Counterfactual Audio-Grounding (CAG) reward that penalizes predictions surviving the removal of acoustic evidence (via silence or corruption), effectively acting as a hallucination penalty; (2) an Attention Density Grounding reward that encourages the model to attend to the specific audio tokens corresponding to the time spans it claims in its output; and (3) a token-wise fine-grained advantage mechanism that assigns credit at the segment level rather than the sequence level, which is crucial for long, dense captions. This approach avoids modifying the base model's architecture or vocabulary, making it a plug-and-play improvement for existing Large Audio Language Models (LALMs). The methodology is sound, addressing the specific failure modes of LALMs on long audio (hallucination and temporal drift) with targeted RL interventions.
The authors introduce EgoSCAPE, a new benchmark derived from Ego4D with 700 manually verified audio clips up to 5 minutes long, featuring dense captions, sound sources, and temporal grounding queries. This is a significant contribution as existing benchmarks often lack dense annotations for long-form audio. Experiments show that ATTEST outperforms strong baselines (including Gemini 3.1 Pro, GPT-4o Audio, and specialized models like TimeAudio and AudioFlamingo 3) on both the new EgoSCAPE benchmark and existing datasets (Clotho-Moment, CASTELLA). The ablation studies convincingly demonstrate the individual contribution of each reward component, particularly the CAG reward in reducing hallucinations. The results on general audio understanding benchmarks (MMAR, LongAudioBench) confirm that the method does not cause catastrophic forgetting, which is a common issue with temporal grounding fine-tuning.
The paper provides a detailed description of the benchmark creation pipeline, including the use of BEATs for initial detection and LLMs for verification, followed by human verification. The RL training details (LoRA, GRPO parameters, reward weights) are specified. However, the code and the EgoSCAPE benchmark itself are not explicitly linked in the provided text (though a reproducibility statement is made). The reliance on specific proprietary models (Gemini, GPT) for baselines limits full reproducibility of those comparisons, but the open-source base models (Qwen, AudioFlamingo) allow for partial reproduction.
The benchmark is limited to 5-minute clips, which is still shorter than some real-world long-form audio scenarios (e.g., hours-long podcasts). The method relies on the availability of attention weights, which may not be accessible in all proprietary or optimized inference engines. The counterfactual reward requires generating a reference model output on corrupted audio, adding computational overhead during training. The paper acknowledges that the method is currently evaluated on audio-only tasks and does not extend to video or omni-modal temporal grounding in the main experiments.
This work addresses a critical bottleneck in deploying LALMs for real-world applications where audio is long and unstructured. By providing a rigorous benchmark (EgoSCAPE) and a training framework (ATTEST) that reduces hallucinations without architectural changes, it enables more reliable temporal reasoning in audio AI. This has implications for accessibility tools, video indexing, and autonomous agents that need to understand and locate specific events in long audio streams. The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
Existing audio deepfake detection (ADD) methods frequently exhibit limited generalization, as fixed bottleneck architectures fail to adapt to the heterogeneous information densities inherent in diverse spoofing artifacts. Moreover, standard augmentation often leads to a pathological robustness-accuracy trade-off. In this paper, we propose a unified ADD framework to address these challenges. Our approach integrates: (1) an Adaptive Audio Feature Integration (AAFI) module that dynamically selects optimal latent dimensions via a compression pool and a stochastic exploration mechanism, ensuring flexible representations across multiple types of dynamic attacks; and (2) a Mel-Band Adversarial Perturbation (Mel-BAP) strategy that applies targeted regularization in the mel-spectrogram domain to encourage the model to learn invariant decision boundaries. Extensive evaluations on benchmarks including ADD 2022, FakeOrReal, and In-the-Wild demonstrate that our model based on Whisper-small achieves a state-of-the-art average Equal Error Rate (EER) of 1.51%. Compared to existing SOTA models such as the 3-billion-parameter Resemble-Detect-3B-Omni, our framework reduces the parameter count by up to 91.6% while achieving a 60.7% improvement in average EER. In a separate cross-lingual evaluation, the Whisper-base configuration achieves EERs of 3.91-5.65% on five unseen languages. The framework also maintains strong resilience to real-world acoustic distortions. This work provides a scalable and efficient "plug-in" solution for enhancing trust and security in the audio Web ecosystem.
Primary: Imperial College London
All Institutions: Imperial College London, Taiyuan University of Technology, University of Southampton, Yangtze Delta Region Institute of Tsinghua University
The paper presents a novel adaptive compression and targeted perturbation framework for audio deepfake detection, achieving state-of-the-art performance with significantly reduced model size. The integration of dynamic feature selection and frequency-domain adversarial training offers a robust solution to the generalization challenges inherent in current ADD methods, providing a scalable and efficient tool for enhancing trust in digital audio content.
The paper proposes a unified framework for Audio Deepfake Detection (ADD) built upon the Whisper-small backbone. The core contribution is the Adaptive Audio Feature Integration (AAFI) module, which employs a heterogeneous compression pool and a stochastic router to dynamically select latent dimensions based on input characteristics. This addresses the limitation of fixed bottleneck architectures by allowing the model to adapt its information density to different types of spoofing artifacts. The second component, Mel-Band Adversarial Perturbation (Mel-BAP), applies targeted adversarial training in the mel-spectrogram domain, specifically focusing on frequency bands where neural vocoder artifacts are prevalent. The methodology is sound, leveraging the strong pre-trained representations of Whisper while introducing adaptive mechanisms to enhance generalization. The use of the Whisper decoder for classification rather than a simple MLP head is a creative choice, justified by the need for temporal granularity and semantic-acoustic alignment.
The experiments are extensive, covering ADD 2022, FakeOrReal, and In-the-Wild benchmarks. The reported state-of-the-art average EER of 1.51% is impressive, especially given the significant reduction in parameter count compared to larger models like Resemble-Detect-3B-Omni. The cross-lingual evaluation on five unseen languages demonstrates strong zero-shot transferability. The ablation studies (referenced in the text) appear to validate the individual contributions of AAFI and Mel-BAP. The comparison against SOTA models is rigorous, highlighting both accuracy and efficiency gains.
The paper provides detailed architectural descriptions, including the formulation of the routing mechanism and the loss functions. However, specific hyperparameters (e.g., noise scale, top-k value, exact compression dimensions) are deferred to the appendix. The reliance on the Whisper backbone ensures that the base model is accessible, but the specific implementation details of the AAFI module and Mel-BAP strategy would require careful replication. The code availability is not explicitly mentioned in the provided text, which is a minor drawback for immediate reproducibility.
The framework relies heavily on the Whisper encoder, which, while powerful, may still struggle with extremely low-quality or heavily distorted audio where the pre-trained features are less reliable. The stochastic exploration mechanism in AAFI adds computational overhead during training, although inference remains efficient. The paper focuses on binary classification (genuine vs. spoofed), and extending this to multi-class or open-set detection scenarios is not explored. Additionally, the performance on real-world, non-benchmarked audio sources is only partially addressed through the In-the-Wild dataset.
This work has significant implications for the security of the audio web ecosystem. By providing a scalable and efficient solution for detecting audio deepfakes, it can be integrated into content moderation systems, verification platforms, and forensic tools. The focus on generalization and robustness makes it particularly relevant for real-world deployment where spoofing techniques evolve rapidly. The efficiency gains also make it suitable for edge devices or large-scale processing, broadening its applicability. The paper presents a novel adaptive compression and targeted perturbation framework for audio deepfake detection, achieving state-of-the-art performance with significantly reduced model size. The integration of dynamic feature selection and frequency-domain adversarial training offers a robust solution to the generalization challenges inherent in current ADD methods, providing a scalable and efficient tool for enhancing trust in digital audio content.
Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing \textbf{UA-Bench}, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0\%--13.1\% leaf accuracy, against a random reference of 7.7\%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.
Primary: Southeast University
All Institutions: Southeast University
The paper introduces UA-Bench, the first large-scale benchmark for evaluating Large Audio-Language Models on underwater acoustic signal recognition, combining parameterized source synthesis with modeled propagation to enable traceable error analysis. By revealing that current LALMs perform near-randomly on fine-grained underwater classification despite some capability in coarse routing, the work provides a rigorous diagnostic framework and a necessary reference point for advancing audio understanding in specialized, non-speech domains.
The methodology is robust and well-designed for a synthetic benchmark. The authors successfully decouple source generation from channel propagation using BELLHOP, allowing for traceable error attribution. The "gated coarse-to-fine" evaluation protocol is a significant methodological contribution, as it distinguishes between failures in source routing (Active vs. Passive) and fine-grained classification, which standard accuracy metrics obscure. The inclusion of a paired silent-audio control is a rigorous diagnostic tool to verify that models are actually utilizing the waveform rather than relying on question priors or class imbalance. The synthesis pipeline, covering detection pulses, communication waveforms, and ship-radiated noise with specific physical parameters (sound-speed profiles, multipath), demonstrates a deep understanding of underwater acoustics.
The experiments are comprehensive, evaluating nine state-of-the-art open-source Large Audio-Language Models (LALMs) without task-specific adaptation. The results are sobering and informative: leaf accuracy ranges from 2.0% to 13.1%, barely above the random reference of 7.7%. The analysis correctly identifies that while some models show significant gains in coarse routing (L1) when audio is present, the fine-grained leaf-level gains are minimal (max +4.7 points). The breakdown of errors into invalid-format, cascade, and classification errors provides valuable insight into the specific failure modes of current LALMs on non-speech audio.
High. The paper provides extensive details on the synthesis parameters, partitioning strategy (source-grouped to prevent leakage), and evaluation protocol. The release of the training partition and the fixed evaluation subset (UA-Bench-Eval) ensures that future work can be compared fairly. The code for generation and evaluation is implied to be available given the nature of the benchmark release, though specific URLs are not in the text provided.
The primary limitation is the synthetic nature of the data. While the propagation is modeled, it may not capture the full complexity of real-world underwater noise (e.g., biological noise, surface clutter, non-linear effects). The paper acknowledges this and positions the benchmark as a controlled reference point rather than a field-ready solution. Additionally, the evaluation is limited to zero-shot performance; the impact of fine-tuning on these models using the provided training partition is not explored in this paper.
This paper provides a critical missing piece in the evaluation of LALMs. By demonstrating that current models struggle significantly with engineered underwater signals, it highlights the gap between general audio pretraining and specialized domain recognition. It sets a baseline for future research in domain adaptation for audio-language models and provides a tool for the maritime and ocean sensing communities to evaluate new models. The paper introduces UA-Bench, the first large-scale benchmark for evaluating Large Audio-Language Models on underwater acoustic signal recognition, combining parameterized source synthesis with modeled propagation to enable traceable error analysis. By revealing that current LALMs perform near-randomly on fine-grained underwater classification despite some capability in coarse routing, the work provides a rigorous diagnostic framework and a necessary reference point for advancing audio understanding in specialized, non-speech domains.
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: https://any-edit.github.io/. AnyEditBench: https://huggingface.co/datasets/Ririiiii/anyeditbench.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Institute of Media Technology and Experience Design (Huawei), City University of Macau, Amphion Technology Co., Ltd.
AnyEdit introduces a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments by decoupling semantic editing from acoustic rendering. The paper makes significant technical contributions, including a noise-robust prosody tokenizer, an instruction-guided autoregressive editor with unmodified-area-aware supervision, and a flow-matching acoustic model for in-context rendering, along with the release of AnyEditBench, a comprehensive benchmark for real-world vocal editing. The methodology is well-motivated, the experiments are thorough, and the results demonstrate state-of-the-art performance, making this a strong and impactful contribution to the field of audio processing.
The paper proposes a unified, separation-free framework for speech and singing voice editing that decouples semantic editing from acoustic rendering. The core innovation lies in the two-stage architecture: an autoregressive (AR) editor that operates in a clean latent space to predict prosody and content-style tokens, followed by a flow-matching transformer (FMT) that renders the final mel-spectrogram conditioned on the source acoustic context. This design elegantly avoids the artifacts associated with source separation while preserving environmental consistency. The introduction of a noise-robust prosody tokenizer via teacher-student distillation is a strong technical contribution, enabling stable prosodic conditioning even from degraded inputs. Additionally, the use of Direct Preference Optimization (DPO) on mined real-to-real singing pairs (leveraging song structure repetitions) is a creative and effective strategy for improving singing quality without relying on synthetic targets. The unmodified-area-aware supervision mechanism is a practical and effective solution to the common problem of prosody drift in localized editing.
The experimental evaluation is comprehensive and rigorous. The authors introduce AnyEditBench, a valuable new benchmark covering speech and singing in clean, simulated, and in-the-wild conditions, which fills a gap in the literature. The evaluation includes both objective metrics (WER/CER, speaker similarity, F0 correlation, background similarity, RT60/SNR error) and subjective evaluations (MOS, Fooling Rate). The results demonstrate consistent superiority over strong baselines, including Vevo2, Step-Audio-EditX, and SSR-Speech, particularly in noisy and reverberant environments. Ablation studies convincingly validate the necessity of each component, such as the noise-robust tokenizer, the staged training, and the DPO alignment. The comparison with SSR-Speech is handled well, with a clear explanation of why SSR-Speech's higher scores on some metrics are not directly comparable due to its full-utterance regeneration nature.
The paper provides sufficient details for reproduction, including model architectures, training strategies, data construction pipelines, and evaluation metrics. The release of AnyEditBench on Hugging Face is a significant contribution to reproducibility and community benchmarking. However, the specific implementation details of the AR editor and FMT are somewhat high-level, and the exact hyperparameters for the DPO training and the teacher-student distillation are not fully detailed in the main text (though likely in the appendix). The use of standard components like Qwen2.5 and LLaMA-2 blocks aids in reproducibility.
The primary limitation is the reliance on a two-stage cascaded architecture, which may introduce latency and potential error propagation from the AR stage to the FMT stage. The model's performance on extremely low SNR conditions (e.g., below 0 dB) is not extensively explored. Additionally, the framework is currently limited to Chinese and English, and its generalization to other languages or more complex polyphonic backgrounds remains to be seen. The DPO mining strategy, while clever, is specific to popular music structures and may not generalize to all types of singing or non-musical vocal performances.
This work has significant potential impact on audio post-production, content creation, and accessibility tools. By enabling high-quality, localized editing of real-world recordings without the need for source separation, it simplifies the workflow for creators and reduces computational overhead. The unified handling of speech and singing opens up new possibilities for music production and interactive media. The release of a comprehensive benchmark will facilitate future research in this area and enable fair comparison of different approaches. AnyEdit introduces a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments by decoupling semantic editing from acoustic rendering. The paper makes significant technical contributions, including a noise-robust prosody tokenizer, an instruction-guided autoregressive editor with unmodified-area-aware supervision, and a flow-matching acoustic model for in-context rendering, along with the release of AnyEditBench, a comprehensive benchmark for real-world vocal editing. The methodology is well-motivated, the experiments are thorough, and the results demonstrate state-of-the-art performance, making this a strong and impactful contribution to the field of audio processing.
Text-conditioned symbolic music generation has attracted growing interest with the development of generative models. Yet two questions remain underexplored: what aspects of music the conditioning text should describe, and how the corresponding data should be collected and trained to improve controllability and text-music alignment. We address these challenges with TASTE, a taxonomy-aware framework for symbolic music generation with a main focus on piano pieces. First, we introduce a taxonomy of musical elements that specifies how a piece can be captured through textual description in a musically grounded way. The taxonomy comprises a set of metadata tags (genre, emotion, section plan, etc.) together with a free-form description of music imagery. This structure decomposes textual input to match the generation goal more coherently and precisely. Second, we build a data pipeline for collecting, annotating, and retrieving public symbolic piano resources, yielding a corpus of 432K MIDI files labeled at multiple levels of granularity. Finally, we design a staged training procedure that leverages this multi-granularity data across two model families: an autoregressive model and a non-autoregressive diffusion model. Both objective and subjective evaluations show that our models achieve superior musical-element control, text-description alignment, and overall music quality. We will release the full pipeline, all annotated datasets, and the model checkpoints.
Primary: University of California, San Diego
All Institutions: University of California, San Diego, University of Rochester
The paper introduces a taxonomy-aware framework for symbolic piano generation that effectively bridges the gap between user intent and model control. By decomposing musical descriptions into structured tags, section plans, and imagery, and building a robust multi-tier data pipeline to annotate 432K MIDI files, the authors demonstrate significant improvements in attribute agreement, temporal structure, and text-music alignment over existing baselines. The rigorous evaluation, including both objective metrics and human listening studies, validates the effectiveness of the proposed staged training procedure for both autoregressive and diffusion models, establishing a new standard for controllable symbolic music generation.
The paper proposes TASTE, a framework that addresses the "description-intention gap" in text-conditioned symbolic music generation. The core methodological contribution is a taxonomy that decomposes musical description into three distinct channels: (1) objective tags (key, tempo, emotion, genre, etc.), (2) section plans (temporal structure), and (3) free-form imagery descriptions. This decomposition allows for targeted annotation and evaluation. The data pipeline is sophisticated, utilizing a multi-tier approach (high-quality human-annotated, curated web-search/LLM-annotated, and large-scale weakly-supervised) to build a corpus of 432K MIDI files. The model architecture integrates these channels into both an Autoregressive (AR) transformer and a Diffusion Transformer (DiT). The AR model uses AdaLN-Zero for tag conditioning and cross-attention for imagery, while the DiT uses similar conditioning mechanisms in a latent space. The staged training procedure, which pretrains on large-scale data with estimated attributes before fine-tuning on high-quality labeled data, is a sound strategy for leveraging weak supervision.
The evaluation is comprehensive, covering objective metrics (tag agreement, section boundary F1, FAD, FMD, Audiobox scores) and subjective listening studies. The results show that TASTE models outperform baselines like MuseCoco, text2midi, and MIDI-LLM in both attribute control and text-music alignment. Notably, the DiT model achieves superior temporal structure realization (boundary F1 of 0.868) compared to the AR model (0.517), suggesting that the non-autoregressive approach handles long-range structural dependencies better. The listening study confirms that listeners can match TASTE outputs to their intended imagery descriptions significantly better than chance and even better than reference performances in some cases, indicating strong semantic alignment.
The paper provides extensive details on the data pipeline, annotation protocols, model architectures, and training configurations. The authors commit to releasing the full pipeline, annotated datasets, and model checkpoints. The use of standard components (Qwen3, CLaMP3, Aria embeddings) and clear descriptions of the weak supervision strategies (probes, retrieval, rewriting) enhance reproducibility. The detailed appendices on probe calibration and confidence gates are particularly helpful for replicating the data construction process.
The primary limitation identified is the reliability of labels for "accompaniment" and "performance style," which remain the hardest fields to control due to lower precision in the weak supervision pipeline. The evaluation is focused on piano music, limiting generalizability to other instruments or full orchestral scores. The reliance on LLMs for annotation (web search and audio description) introduces potential biases and inconsistencies, although mitigated by the multi-tier verification process. The AR model's weaker performance in temporal structure realization compared to DiT is a notable weakness for that specific architecture.
This work contributes to the field of controllable music generation by providing a principled framework for describing and controlling musical attributes. The taxonomy and data pipeline can be extended to other musical domains. The release of a large, multi-granularity annotated dataset will benefit the research community by enabling fairer comparisons and more fine-grained control in future models. The integration of symbolic and perceptual conditioning offers a path toward more human-like interaction with music generation systems. The paper introduces a taxonomy-aware framework for symbolic piano generation that effectively bridges the gap between user intent and model control. By decomposing musical descriptions into structured tags, section plans, and imagery, and building a robust multi-tier data pipeline to annotate 432K MIDI files, the authors demonstrate significant improvements in attribute agreement, temporal structure, and text-music alignment over existing baselines. The rigorous evaluation, including both objective metrics and human listening studies, validates the effectiveness of the proposed staged training procedure for both autoregressive and diffusion models, establishing a new standard for controllable symbolic music generation.
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.
Primary: Cornell University
All Institutions: Cornell University
The paper introduces a novel synthetic data generation pipeline for ultrasonic silent speech recognition, significantly improving WER through a two-stage GAN and cross-modal contrastive learning. The rigorous ablations and controls demonstrate the specific value of the synthetic data, establishing a new state-of-the-art for minimally obtrusive silent speech interfaces.
The paper proposes SoniGAN, a two-stage generative framework for synthesizing ultrasonic echo profiles from text. The methodology is technically sound and well-motivated. It leverages a voice-cloned TTS model (F5-TTS) to generate audio from text, converts this to soft speech units to strip timbre, and then uses a conditional GAN architecture to map these units to ultrasonic signals. The two-stage approach (envelope first, then fringe structure) is a clever solution to the high-frequency nature of ultrasonic fringes, which are difficult for single-step generators to capture. The use of a frozen recognizer as a perceptual loss is a strong choice to ensure semantic fidelity. The inclusion of cross-modal contrastive learning (crossCon) in the recognizer training is a significant methodological contribution that effectively utilizes the paired voiced/silent data available in the benchmark.
The experiments are rigorous and comprehensive. The authors provide a clear ablation study showing the contribution of each component (synthetic pretraining, faulty session handling, decoding strategies). The controls for "extra optimizer steps" vs. "data content" are particularly well-designed, proving that the gain comes from the synthetic data's content rather than just increased training time. The results show a significant improvement in Word Error Rate (WER), moving from a baseline of ~26% to 14.9%, which is a substantial leap for this modality. The evaluation on the public SoniSpeech benchmark ensures comparability.
The paper provides detailed hyperparameters, architecture descriptions, and training procedures. However, the specific implementation of the "soft speech units" extraction and the exact configuration of the F5-TTS cloning process are not fully detailed, which might pose challenges for exact reproduction. The code is not explicitly linked in the provided text, though the use of standard components (ResNet, Transformer, F5-TTS) aids in reconstruction.
The system relies on a specific benchmark (SoniSpeech) and a specific TTS model. The generalizability to other ultrasonic sensing setups or different languages is not tested. The "faulty microphone" augmentation is specific to the hardware failure mode of the benchmark device. The 14.9% WER, while a major improvement, is still higher than typical audible speech recognition systems, indicating room for further improvement.
This work has significant implications for accessibility and human-computer interaction. Silent speech interfaces are crucial for individuals with vocal impairments or for use in noisy environments. By reducing the data collection burden via synthetic supervision, this approach makes it easier to scale silent speech systems to open vocabularies, potentially enabling more natural and less obtrusive communication interfaces. The paper introduces a novel synthetic data generation pipeline for ultrasonic silent speech recognition, significantly improving WER through a two-stage GAN and cross-modal contrastive learning. The rigorous ablations and controls demonstrate the specific value of the synthetic data, establishing a new state-of-the-art for minimally obtrusive silent speech interfaces.
Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.
Primary: MIT (Inferred from "MIT-IBM Watson AI Lab" and common affiliation patterns for this research group; specific author list missing in text but "MIT-IBM" is explicit)
All Institutions: MIT, IBM Research, Hyundai Motor Company (Supporters)
The paper demonstrates that harmonic decisions in a symbolic music transformer are causally localized in middle layers and can be manipulated via latent steering to control cadence resolution. By combining causal interventions, linear probing, and representation steering on Bach chorales, the authors reveal that the model's internal geometry mirrors music theory (with relative keys closer than parallel keys) and that these representations are actionable for inference-time control, offering a new pathway for interpretable and controllable symbolic music generation.
The paper employs a rigorous mechanistic interpretability framework applied to a symbolic music transformer (Aria). The methodology is sound, utilizing three complementary techniques: (1) Causal interventions (activation patching/interchange interventions) to localize harmonic computation, (2) Linear probing to assess decodability of harmonic attributes (mode, key), and (3) Representation steering to test actionability. The construction of counterfactual pairs via minimal single-note edits to Bach chorales is a clever and effective way to isolate specific harmonic decisions (cadence resolution) from other musical features. The derivation of steering vectors from probe weights is a standard but effective technique in this domain. The analysis of the geometric relationship between relative and parallel keys in the latent space provides a strong theoretical link between model internals and music theory.
The experiments are well-designed and comprehensive. The use of Bach chorales is appropriate for this specific query (harmonic structure), though it limits generalizability. The evaluation metrics are robust, combining structural integrity checks (error rates, repetition), perceptual quality (FMD), and harmonic specificity (Roman numeral entropy, top-1 dominance). The results clearly demonstrate that harmonic representations are localized in middle layers and that steering these representations effectively alters the harmonic outcome (e.g., shifting from authentic to deceptive cadences) without degrading overall musical coherence. The finding that parallel key steering is more effective than relative key steering due to latent space geometry is a significant and insightful result.
The paper provides sufficient detail for reproducibility. It specifies the model (Aria), the dataset (Bach chorales), the tool for harmonic analysis (music21), and the specific intervention methods (layers, alpha values, probe training details). The code and data are available on the project webpage. The specific layer ranges for intervention (5-10 for mode, 11-15 for key) are clearly stated.
The primary limitation is the narrow scope of the dataset (Bach chorales). The authors acknowledge that whether these findings hold for more chromatic or less conventional music is an open question. The model used (Aria) is a specific LLaMA-based transformer; results may not generalize to other architectures (e.g., diffusion models for audio, or different tokenization schemes). The steering is limited to harmonic resolution at cadences; it is unclear if this method can control other musical aspects like melody or rhythm with similar precision.
This work contributes to the growing field of mechanistic interpretability in generative models, specifically for music. It demonstrates that high-level musical concepts are not just distributed noise but are structured and manipulable. This has implications for controllable music generation, allowing users to steer harmonic progressions without retraining. It also provides insights into how neural networks internalize symbolic rules, bridging the gap between music theory and machine learning. The paper demonstrates that harmonic decisions in a symbolic music transformer are causally localized in middle layers and can be manipulated via latent steering to control cadence resolution. By combining causal interventions, linear probing, and representation steering on Bach chorales, the authors reveal that the model's internal geometry mirrors music theory (with relative keys closer than parallel keys) and that these representations are actionable for inference-time control, offering a new pathway for interpretable and controllable symbolic music generation.
High-fidelity audio coding requires faithful reconstruction of perceptually important high-frequency components, which are often lost due to the aggressive temporal decimation and quantization of neural codecs. State-of-the-art neural audio coding methods rely on adversarial learning to hallucinate these missing components, yielding perceptually plausible but imprecise reconstructions; a limitation for high-fidelity music applications. As a remedy, we propose high-efficiency neural audio coding (HE-NAC), which provides a multi-band extension of the popular residual vector quantization (RVQ) scheme. HE-NAC introduces an effective training mechanism that harmonizes a bitrate-free blind upsampling process, residual learning to compensate it with minimal bitrate, and a multi-head decoder architecture for band-specific reconstruction. As a result, HE-NAC achieves a balanced trade-off, reliably reconstructing the core-band ($\leq$ 3 kHz) component while preserving overall sound fidelity. Subjective listening tests and objective evaluations demonstrate that HE-NAC outperforms both the DAC baseline and MPEG HE-AAC at equal or lower bitrates.
Primary: Electronics and Telecommunications Research Institute (ETRI)
All Institutions: Indiana University, Electronics and Telecommunications Research Institute (ETRI), University of Illinois at Urbana-Champaign
HE-NAC introduces a multi-band extension of RVQ that effectively balances core-band fidelity with high-frequency reconstruction efficiency. The paper presents a well-motivated solution to the high-frequency loss problem in neural audio codecs, leveraging insights from traditional SBR algorithms within a modern deep learning framework. The combination of architectural innovation (multi-band RVQ) and a tailored training strategy (harmonizing blind upsampling with residual learning) results in a codec that demonstrates superior performance over state-of-the-art neural baselines and commercial standards, marking a significant step toward practical high-fidelity neural audio coding.
The paper proposes HE-NAC, a neural audio codec that extends Residual Vector Quantization (RVQ) into a hierarchical multi-band framework. The core methodological contribution is the decoupling of the core band (≤ 3 kHz) and high-frequency bands, allowing for independent bitrate allocation. The architecture utilizes a "blind upsampling" process for high-frequency reconstruction, which is then refined by residual learning to minimize the error introduced by the upsampling. This is inspired by traditional SBR (Spectral Band Replication) algorithms but implemented via deep learning. The training strategy is multi-stage, aiming to harmonize the blind upsampling with the residual coders to ensure perceptual quality in the core band while using minimal bits for high-frequency details. The approach is technically sound and addresses a known limitation of single-scale neural codecs (loss of high-frequency fidelity due to aggressive temporal decimation).
The evaluation includes both objective metrics and subjective listening tests (MUSHRA-like). The paper claims that HE-NAC outperforms the DAC (Descript Audio Codec) baseline and MPEG HE-AAC at equal or lower bitrates (specifically 23 kbps vs 24 kbps for HE-AAC). The comparison to a commercial standard (HE-AAC) is a strong point, as it demonstrates practical viability. However, the paper text provided is truncated, so the specific numerical values for objective metrics (like SI-SDR, PESQ, or ViSQOL) and the detailed breakdown of subjective scores are not fully visible in the excerpt, though the abstract confirms the superiority.
The authors provide a GitHub repository link (https://github.com/darius522/henac), which significantly enhances reproducibility. The architectural details (multi-head decoder, RVQ extension) are described in sufficient detail for implementation. The multi-stage training algorithm is a key component that would require careful tuning, but the open-source code mitigates this risk.
The paper focuses on monoaural audio coding. It does not address stereo or spatial audio, which is a growing area of interest. The computational complexity of the multi-band architecture and the multi-stage training process may be higher than simpler single-scale codecs, though this is typical for high-fidelity neural codecs. The reliance on "blind upsampling" still involves some generative hallucination, which the paper acknowledges as a trade-off, though it mitigates it with residual coding.
This work has significant implications for high-fidelity music streaming and storage, where perceptual quality is paramount. By achieving quality comparable to or better than HE-AAC at lower bitrates using neural methods, it could influence future codec standards and streaming services. The integration of traditional SBR concepts with modern neural RVQ provides a bridge between legacy audio coding knowledge and deep learning advancements. HE-NAC introduces a multi-band extension of RVQ that effectively balances core-band fidelity with high-frequency reconstruction efficiency. The paper presents a well-motivated solution to the high-frequency loss problem in neural audio codecs, leveraging insights from traditional SBR algorithms within a modern deep learning framework. The combination of architectural innovation (multi-band RVQ) and a tailored training strategy (harmonizing blind upsampling with residual learning) results in a codec that demonstrates superior performance over state-of-the-art neural baselines and commercial standards, marking a significant step toward practical high-fidelity neural audio coding.
Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.
Primary: AutoArk
All Institutions: AutoArk
The paper presents Audio8-TTS, a compact 0.6B parameter zero-shot voice cloning model that achieves state-of-the-art perceptual naturalness by employing a reference-based realism critic trained on a quality ladder of open-source TTS systems, integrated into a Group Relative Policy Optimization (GRPO) framework to optimize prosodic realism alongside intelligibility and speaker similarity.
The paper proposes a three-stage training framework for a compact (0.6B) dual-autoregressive TTS model. The core novelty lies in the third stage: using a learned "realism critic" as a reward model for Group Relative Policy Optimization (GRPO). The critic is trained on a "quality ladder" constructed from outputs of various open-source TTS systems, ranked via a private blind listening study, with human recordings as the top anchor. A key technical contribution is the "identity-erasure" objective, which uses pause-compression degradation to prevent the critic from relying on generator-specific fingerprints rather than prosodic quality. The architecture itself (fast-slow dual autoregressive) is standard in recent codec-LM literature, but the specific application of reference-based realism rewards to compact models is a distinct methodological contribution.
The evaluation is comprehensive, covering 9 CV3 languages and 13 additional languages. The model achieves competitive WER and SIM scores compared to much larger models (e.g., Seed-TTS, MiniMax). The most significant evidence is the blind human evaluation, where the realism-optimized model received a substantially higher Elo rating than the SFT and WER/SIM-only models, validating the effectiveness of the realism critic. The ablation of the critic's generalization to unseen systems (Fish Audio S2, Spark-TTS) is a strong point, showing the critic learns quality rather than memorizing specific generators.
The authors release code and model weights on GitHub. They provide detailed descriptions of the data mixture, training stages, and critic construction. However, the "private blind listening study" and "private collection" of data limit full reproducibility of the critic's training data. The use of specific open-source systems for the quality ladder is reproducible, but the exact ranking and degradation parameters are proprietary.
The realism critic requires a matched human reference for every sentence during inference/evaluation, which is a significant practical constraint for zero-shot cloning where such references may not exist. The paper acknowledges that the critic's signal is primarily driven by pause structure due to the degradation method used, potentially neglecting other prosodic cues like pitch and energy. Additionally, the model struggles with low-resource languages (e.g., Thai, Finnish) where error rates remain high.
This work demonstrates that compact TTS models can achieve high perceptual naturalness through targeted preference optimization, not just scale. The approach of using a learned critic to bridge the gap between objective metrics (WER/SIM) and subjective realism is applicable to other generative audio tasks. The identity-erasure technique is a valuable contribution to the field of reward model design for generative models. The paper presents Audio8-TTS, a compact 0.6B parameter zero-shot voice cloning model that achieves state-of-the-art perceptual naturalness by employing a reference-based realism critic trained on a quality ladder of open-source TTS systems, integrated into a Group Relative Policy Optimization (GRPO) framework to optimize prosodic realism alongside intelligibility and speaker similarity.
Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.
Primary: Brown University
All Institutions: Brown University, MIT, Yale University, The University of Hong Kong
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Sonic-AMG introduces a novel framework for streaming human motion generation from spatial audio, addressing key limitations of prior offline methods. The core methodology is built upon a causal transformer with a flow head, operating on continuous motion latents from a pretrained VAE. A significant methodological contribution is the separation of acoustic content ("what") and body-relative source geometry ("where") into distinct conditioning tokens. This design choice, supported by ablations, allows the model to better disentangle and utilize these modalities. The 11-dimensional spatial descriptor, inspired by perceptual research, is a thoughtful design for capturing relevant spatial cues like proximity, relative motion, and looming. The paper effectively tackles two critical training-inference mismatches inherent in autoregressive streaming generation. First, the "history mismatch" (teacher-forced training vs. generated histories at inference) is addressed by "rehearsal training." This innovative approach combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. This hybrid strategy allows the model to learn stable long-range dependencies while also gaining robustness to accumulated errors from its own predictions. Second, the "spatial conditioning mismatch" (ground-truth SSL vs. generated root poses) is resolved by "spatial feedback." This mechanism recomputes body-relative source locations from generated root poses during both rehearsal training and inference, ensuring consistency between the generated motion and the spatial conditions. The integration of these components within a causal transformer and flow-matching framework is technically sound and well-justified. The use of classifier-free guidance during inference further enhances generation quality.
The experimental evaluation is comprehensive and well-structured, primarily using the SAM dataset. The paper establishes state-of-the-art performance for Sonic-AMG on FID and R-precision metrics, demonstrating both distributional fidelity and conditioning alignment. A crucial aspect of the evaluation is the distinction between "GT SSL" (ground-truth spatial conditioning, for fair comparison with baselines) and "Online SSL" (spatial feedback from generated motion, reflecting real-world streaming). Sonic-AMG maintains strong performance under the more challenging Online protocol, which is a significant achievement. The ablation studies are particularly strong, providing clear evidence for the efficacy of the proposed components: 1. **Rehearsal training vs. two-forward training:** Rehearsal training consistently outperforms various two-forward schedules, validating the benefit of recursive history construction. 2. **Role of teacher-forced supervision:** Demonstrates that combining self-rollouts with full-sequence supervision is crucial for fidelity and alignment. 3. **Effect of spatial feedback:** Shows that training with spatial feedback dramatically improves performance under the Online protocol, confirming its necessity for consistent spatial conditioning. 4. **Separate spatial tokens:** Ablation against a "Fused" approach confirms the advantage of distinct acoustic and spatial tokens. 5. **Learned acoustic features:** Shows that simpler, learned log-mel features outperform MOSPA's high-dimensional audio descriptors, supporting the hypothesis that separating "what" from "where" helps the acoustic branch focus on content. Beyond quantitative metrics, the paper provides compelling qualitative results, including examples of recognizable and spatially appropriate reactions, and the ability to generate coherent motion across complete audio clips and successive sound events. The demonstration of real-time streaming efficiency, with significantly reduced initial response latency compared to MOSPA, is a strong practical contribution. The inclusion of physical simulation retargeting to a Unitree G1 robot further showcases the robustness and applicability of the generated motions. The "No-SSL" variant, which derives spatial tokens directly from binaural audio, is an excellent exploration of alternative inference scenarios, demonstrating the model's flexibility and potential for deployment without explicit source location inputs.
The paper includes a dedicated "Reproducibility statement" and detailed appendices covering implementation details, training protocols, and runtime measurements. This includes specifics on architecture (transformer layers, heads, width, flow head details), optimization (AdamW, learning rate, batch size), training hardware and time, motion autoencoder pretraining, acoustic and spatial encoding, guidance, and training sequence construction for both teacher-forced and rehearsal training. The authors commit to releasing code and models upon acceptance, which, combined with the extensive details provided, suggests high reproducibility.
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Primary: FAIR, Meta Superintelligence Labs
All Institutions: The University of Texas at Austin, FAIR, Meta Superintelligence Labs
The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
The paper proposes RUBATO, a multistream positional encoding mechanism that assigns independent "clocks" to different data streams (real-time audio vs. internal thinking) in full-duplex speech models. The core innovation is decoupling the positional indices of asynchronous streams to prevent temporal distortion when the decoding rate of the thinking stream varies. The authors derive three variants: TM-RoPE (subspace partitioning), Key-Clock (conditional rotation based on key token clock), and Full-Clock (rotating all dimensions by the key's clock). The methodology is mathematically sound, leveraging the properties of Rotary Position Embeddings (RoPE) to allow zero-shot interpolation of inference speeds. The implementation strategy, which expands hidden dimensions during training to fit standard FlashAttention kernels and uses separate KV caches during inference, is a clever engineering solution that avoids custom Triton kernels, enhancing practicality.
The experiments are rigorous, evaluating the model on Spoken GSM8K and BIG-Bench Audio, including a novel "Overlapped" scenario where the model must handle interruptions while reasoning. The results demonstrate that RUBATO-Full maintains high accuracy (48-51%) across varying decoding budgets, whereas baselines collapse to near-zero accuracy at interpolated speeds. The paper provides strong evidence for the utility of the method in reducing time-to-answer by leveraging faster hardware without retraining. The inclusion of UTMOS and Speaker Similarity metrics for filler audio quality adds depth to the evaluation, confirming that the architectural changes do not degrade acoustic quality.
The paper provides a detailed reproducibility statement, including closed-form equations for the attention mechanisms and clear descriptions of the training data synthesis process. The use of standard backbones (Gemma-4) and tokenizers (NanoCodec) aids reproducibility. However, the specific hyperparameters for the training mixture and the exact construction of the synthetic datasets are deferred to the appendix, which is not fully visible in the provided text, though the main text describes the logic clearly.
The primary limitation is the reliance on synthetic data for training, as no suitable full-duplex spoken reasoning corpora exist. The evaluation is limited to mathematical and logical reasoning tasks (GSM8K, BIG-Bench), which may not generalize to all conversational domains. Additionally, the method assumes a specific interleaving strategy (SCoT), and its applicability to other full-duplex architectures (e.g., concatenation-based) is not explored. The "Full-Clock" variant, while best performing, requires duplicating the head dimension, which increases memory usage.
This work has significant implications for the deployment of real-time AI agents. By decoupling reasoning speed from real-time constraints, it enables more efficient use of heterogeneous hardware. The concept of independent clocks for asynchronous streams is a generalizable architectural pattern that could apply to other multimodal or multi-agent systems requiring variable-rate processing. It addresses a critical bottleneck in making LLMs truly interactive and responsive in speech-based interfaces. The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.
Primary: Eindhoven University of Technology
All Institutions: Eindhoven University of Technology
AudioTETRIS introduces a training-free token compression framework for LALMs that leverages encoder-state dynamics as a phoneme-transition prior to guide attention-based pruning. The paper demonstrates that preserving acoustically rich transition regions is crucial for maintaining high accuracy under aggressive compression, outperforming existing baselines across multiple backbones and tasks while significantly reducing inference compute.
The paper proposes AudioTETRIS, a training-free token compression framework for Large Audio Language Models (LALMs). The core innovation is the derivation of a "phoneme-transition-aware" acoustic prior from the temporal derivative of the audio encoder's hidden states. The authors hypothesize that regions of rapid spectral transition in the encoder space correspond to phonetic transitions, which carry high information density. This prior is fused with a depth-adaptive attention score from the LLM decoder to guide progressive token pruning. The methodology is sound, leveraging existing model components (encoder hidden states, decoder attention) without requiring fine-tuning. The validation of the acoustic prior via correlation with forced alignments and kurtosis analysis is a strong methodological addition that grounds the heuristic in linguistic reality.
The experiments are comprehensive, evaluating three distinct backbones (Qwen2-Audio-7B, Phi-4-Multimodal, Audio Flamingo 3) across four diverse tasks (ASR, Emotion Recognition, Audio QA, Translation). The results demonstrate that AudioTETRIS retains near-lossless performance at moderate compression rates and significantly outperforms baselines (SpeechPrune, SparseVLM, LTBM) at aggressive compression rates (75% token removal). The ablation studies effectively isolate the contributions of the acoustic prior and the attention score, confirming their complementary nature. The efficiency analysis provides concrete FLOP and latency savings, validating the practical utility of the method.
The paper provides high reproducibility standards. It includes detailed algorithmic summaries, specifies the exact layers for pruning and prior computation, and describes the calibration procedure for the crossover layer. The authors commit to releasing the full implementation and evaluation scripts. The use of standard, publicly available datasets and checkpoints further enhances reproducibility.
The method requires a one-time calibration pass per backbone to determine the crossover layer, which limits its "zero-shot" applicability to unseen architectures. The acoustic prior assumes specific geometric properties of encoder hidden states that may not hold for all encoder types (e.g., HTSAT-style event encoders). The evaluation is limited to English-source benchmarks, and performance on noisy or multilingual audio remains untested. The progressive pruning within the decoder incurs a slight TFLOP overhead compared to single-shot prefill pruning, though this is offset by accuracy gains.
This work addresses a critical bottleneck in deploying LALMs: the quadratic scaling of self-attention with long audio token sequences. By providing a training-free, plug-and-play solution that preserves fine-grained acoustic details, AudioTETRIS enables more efficient and cost-effective deployment of speech-text models. The insight that encoder representation geometry can serve as a structural prior for compression is broadly applicable to other multimodal settings where modality-specific encoders produce redundant token sequences. AudioTETRIS introduces a training-free token compression framework for LALMs that leverages encoder-state dynamics as a phoneme-transition prior to guide attention-based pruning. The paper demonstrates that preserving acoustically rich transition regions is crucial for maintaining high accuracy under aggressive compression, outperforming existing baselines across multiple backbones and tasks while significantly reducing inference compute.
Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.
Primary: Seoul National University
All Institutions: Seoul National University
LiveSynth introduces a streaming neural synthesizer capable of real-time zero-shot instrument cloning and text-to-instrument synthesis with 10 ms latency. By leveraging a feedback-free causal Transformer and frame-wise timbre conditioning, it achieves competitive quality with offline baselines while enabling interactive features like continuous timbre morphing and autonomous performance continuation, marking a significant step toward playable generative audio models.
The paper proposes LiveSynth, a streaming neural synthesizer that adapts the feedback-free causal Transformer framework from LiveBand for instrument cloning and text-to-instrument synthesis. The core methodological contribution is the integration of frame-wise timbre conditioning (using CLAP embeddings) with a causal VAE decoder operating at 100 Hz. The use of MIDI-conditioning dropout to enable autonomous continuation from a performed prefix is a clever training trick that facilitates interactive "call-and-response" scenarios. The decision to use continuous latents rather than quantized tokens is well-justified by the need for low latency and high reconstruction quality, avoiding the error accumulation and quality degradation associated with autoregressive token generation in strict causal settings. The Procrustes alignment for text-to-audio embedding mapping is a practical and effective solution to the modality gap in CLAP embeddings.
The experimental evaluation is rigorous, comparing LiveSynth against strong offline baselines (TokenSynth, CTD) on both NSynth and Slakh datasets. The metrics used (MERT, PANNs, MSS, F1, CE, PQ) provide a comprehensive view of timbre similarity, spectral quality, and perceptual aesthetics. The real-time inference benchmark on an M3 MacBook Air is highly relevant and convincing, demonstrating an RTF of 0.88 with 10 ms latency, which is critical for the "playable instrument" claim. The ablation on timbre morphing and autonomous continuation provides good evidence for the specific interactive capabilities. However, the lack of human listening tests (MOS/ABX) is a significant gap for a paper claiming perceptual quality improvements.
The paper provides detailed architectural specifications (parameter counts, hop sizes, attention windows) and training hyperparameters. The code and demo are available via the provided URL. The use of standard datasets (NSynth, Slakh) and open-source components (CLAP, Vocos, DAC) enhances reproducibility. However, the specific implementation of the Procrustes alignment and the exact configuration of the adversarial training might require careful attention to replicate exactly.
The primary limitation is the lack of subjective human evaluation (listening tests), which is standard for audio generation papers. The paper acknowledges lower note accuracy (F1) and production quality (PQ) compared to some offline baselines, which is a trade-off for real-time performance. The rhythmic dependence in autonomous continuation is weak, as noted by the high IOI-JSD distance. The system is currently limited to mono audio and specific instrument types found in the training data.
This work bridges the gap between offline generative audio models and real-time interactive music tools. It enables new forms of human-AI musical interaction, such as live jamming with AI-generated instruments that can morph timbres or continue performances autonomously. This has implications for music production, education, and performance art. The streaming architecture could be extended to other domains requiring real-time conditional generation. LiveSynth introduces a streaming neural synthesizer capable of real-time zero-shot instrument cloning and text-to-instrument synthesis with 10 ms latency. By leveraging a feedback-free causal Transformer and frame-wise timbre conditioning, it achieves competitive quality with offline baselines while enabling interactive features like continuous timbre morphing and autonomous performance continuation, marking a significant step toward playable generative audio models.
Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.
Primary: Tencent (Hunyuan)
All Institutions: Tencent (Hunyuan), Nanyang Technological University, Northumbria University
The paper introduces OmniCapBench, a deep-structured benchmark that reframes audio-visual captioning evaluation by decomposing free-form text into atomic, verifiable units (entities, shots, events), enabling reliable, fine-grained diagnostic scoring that exposes specific model failures like identity drift and temporal grounding errors which holistic metrics obscure.
The paper proposes a "deep-structured" evaluation framework that shifts the paradigm from scoring free-form text captions to scoring structured, atomic JSON units (References, Shots, Events). This is a methodologically sound approach to solving the "opaque score" problem in LLM-as-a-Judge evaluations. By decoupling structural verification (deterministic rules for IDs, timestamps, and links) from semantic verification (bounded LLM checks on aligned units), the authors effectively mitigate the instability and hallucination sensitivity of global LLM judges. The construction pipeline, which uses a multi-model ensemble (Generator, Evaluator, Refiner, Selector) with strict schema validation, is robust and well-designed to ensure ground-truth integrity. The bidirectional evaluation (Precision vs. Recall) for open-vocabulary fields is a clever addition that prevents models from gaming the metric via verbosity or terseness.
The experimental section is rigorous and provides strong evidence for the benchmark's utility. The authors evaluate 9 state-of-the-art omnimodal models, distinguishing between proprietary and open-source variants. The results clearly demonstrate the benchmark's ability to differentiate models that might appear similar under holistic metrics. The "Auditing the Legacy Evaluation Paradigm" section is particularly strong, providing empirical proof that global LLM judges are unstable (scores varying by >20 points based on judge strength) and that holistic scores mask local errors. The error analysis, decomposing failures into specific structural violations (e.g., identity drift, temporal grounding), offers actionable insights for model developers.
The paper provides high reproducibility. It details the exact JSON schema, the filtering thresholds for data curation, the temperature settings for inference, and the specific prompts used for generation and judging (referenced in the appendix). The project website is provided, which likely hosts the dataset and code. The clear separation of deterministic rules from LLM-based scoring makes the evaluation pipeline easy to replicate and debug.
The primary limitation is the reliance on structured JSON output for evaluation. This may not reflect the natural usage of MLLMs, which are often deployed for free-form conversation or captioning. Forcing models to output strict JSON can introduce a separate mode of failure (formatting errors) that is distinct from perceptual errors, although the authors do account for this with the SGC (Schema Gate Check) metric. Additionally, the benchmark is limited to videos under 5 minutes, which may not capture long-horizon reasoning capabilities in longer films or lectures. The annotation process, while rigorous, is still automated/semi-automated, and potential biases in the "Evaluator" models used to construct the ground truth could propagate into the benchmark.
This benchmark has significant potential to standardize the evaluation of omnimodal models. By providing a fine-grained diagnostic tool, it helps researchers identify specific bottlenecks (e.g., cross-modal alignment vs. entity tracking) rather than just overall performance. This granularity is crucial for the next generation of audio-visual AI, where understanding the *relationship* between sound and sight is key. The framework could be extended to other multimodal tasks (e.g., video QA, action recognition) by adapting the atomic unit structure. The paper introduces OmniCapBench, a deep-structured benchmark that reframes audio-visual captioning evaluation by decomposing free-form text into atomic, verifiable units (entities, shots, events), enabling reliable, fine-grained diagnostic scoring that exposes specific model failures like identity drift and temporal grounding errors which holistic metrics obscure.
Group dance generation aims to synthesize coordinated multi-dancer choreography from music, with broad applications in animation and interactive content creation. This task requires modeling dense inter-person dependencies to ensure spatial coordination, while naturally preserving individual dancer identities. Existing approaches model all dancers jointly with end-to-end transformers, which tie the architecture to a fixed group size and entangle per-dancer identities across frames. We propose ChainDance, a scalable framework that reformulates group dance generation as a Chain-of-Dancers: a sequential decomposition over per-dancer conditional distributions, allowing a single model to scale across variable group sizes without retraining and naturally preserving per-dancer identity. Built on a frozen single-dancer diffusion backbone, ChainDance introduces two lightweight modules: a Role-Aware Text Encoder (RATE) for per-dancer semantic conditioning, and a Group-Aware Motion Encoder (GAME) that aggregates previously generated dancers via a distance-weighted graph convolutional network, and incorporates a training-free noise optimization procedure at inference time to enforce global spatial coherence. Experiments on AIOZ-GDance demonstrate that ChainDance achieves state-of-the-art motion quality and group coordination while structurally preserving per-dancer identity, with $3$-$4\times$ fewer parameters and requiring $3$-$6\times$ less training time compared to prior approaches.
Primary: Monash University
All Institutions: Monash University
The paper introduces a scalable and parameter-efficient framework for group dance generation by reformulating the problem as a sequential chain of conditional single-dancer generations, effectively addressing the rigidity and identity entanglement issues of joint modeling approaches.
The paper proposes ChainDance, a framework that reformulates group dance generation as a sequential "Chain-of-Dancers" rather than a joint multi-agent generation task. This is a sound architectural decision that addresses the rigidity of fixed group sizes in transformer-based joint models. The method leverages a frozen single-dancer diffusion backbone, injecting group awareness via two lightweight modules: RATE (Role-Aware Text Encoder) for semantic conditioning and GAME (Group-Aware Motion Encoder) for spatial/kinematic conditioning using a distance-weighted GCN. The use of LoRA for parameter efficiency is appropriate. The inference-time optimization (PINO-based) to enforce global coherence is a clever workaround for the lack of global attention in the sequential generation process, though it introduces computational overhead. The decomposition into pairwise leader-follower interactions is a reasonable approximation of the full joint distribution, though it may struggle with complex non-pairwise dynamics (e.g., three-person interactions).
The paper claims state-of-the-art performance on the AIOZ-GDance dataset with significant reductions in parameters and training time. However, the provided text is truncated before the detailed results tables, making it difficult to verify the magnitude of improvements or the specific metrics used (e.g., FID, FVD, MPJPE, collision rate). The claim of "3-4x fewer parameters" is strong but needs verification against baseline architectures. The reliance on a single dataset (AIOZ-GDance) limits the generalizability of the claims. The use of ChatGPT-4o for instruction generation is a practical choice but introduces a dependency on external LLMs for data preparation, which may not be reproducible for all users.
The paper provides high-level algorithmic descriptions but lacks specific hyperparameters, learning rates, and detailed implementation choices for the GCN and LoRA modules in the visible text. The use of a "frozen single-dancer diffusion backbone" is not specified (which model? e.g., MoDi, DMD?), which is critical for reproducibility. The rule-based detection for interaction keywords is mentioned but not detailed. The absence of code links or a project page in the extracted text reduces reproducibility.
The sequential generation approach may suffer from error accumulation, where errors in early dancers propagate to later ones. The pairwise approximation may fail to capture higher-order group dynamics. The inference-time optimization adds latency, which may be prohibitive for real-time applications. The method relies on a pre-trained single-dancer model, so its performance is upper-bounded by the quality of that backbone. The use of LLMs for instruction generation may introduce biases or inconsistencies in the training data.
The work has significant implications for scalable motion generation in animation, VR, and interactive media. By decoupling group size from model architecture, it enables flexible deployment in applications where the number of agents varies. The parameter efficiency makes it more accessible for researchers and developers with limited computational resources. The approach could be extended to other multi-agent generation tasks, such as crowd simulation or multi-robot coordination. The paper introduces a scalable and parameter-efficient framework for group dance generation by reformulating the problem as a sequential chain of conditional single-dancer generations, effectively addressing the rigidity and identity entanglement issues of joint modeling approaches.
Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.
Primary: Xiaomi Inc.
All Institutions: Xiaomi Inc.
MiDashengLM-Spatial unifies general audio understanding and spatial awareness in a single open-source model. The paper presents a rigorous integration of spatial and semantic encoders via hierarchical conditioning, supported by a large-scale synthetic data pipeline and comprehensive evaluations that demonstrate the model's ability to perform spatial reasoning without compromising general audio capabilities.
The paper proposes MiDashengLM-Spatial, a unified architecture that integrates a monaural semantic encoder (Dasheng) and a spatial encoder (Spatial-Dasheng) via a Hierarchical Semantic-to-Spatial Conditioning (HSSC) module. The HSSC mechanism is a key architectural contribution, injecting intermediate semantic features into the spatial branch at multiple depths to facilitate the association of spatial cues with semantic content, particularly for overlapping sources. The spatial encoder itself is built on a Vision Transformer (ViT) backbone pre-trained with a Masked Autoencoder (MAE) objective, adapted for multi-channel input via a lightweight CNN fusion module. The training paradigm uses a multi-ACCDOA output format with Permutation-Invariant Training (PIT) to handle overlapping sources. The data pipeline is robust, synthesizing 1 million binaural clips and 6 million QA pairs using room impulse responses (RIRs) and LLM-generated scene descriptions, ensuring ground-truth alignment between spatial metadata and language.
The experimental evaluation is comprehensive. Spatial-Dasheng is evaluated on standard SELD benchmarks (MRSSound, EasyCom, STARSS23), showing strong performance and sim-to-real generalization. MiDashengLM-Spatial is evaluated on spatial reasoning benchmarks (MMAU-Pro, STAR-Bench) and general audio benchmarks (ASR, Captioning, QA). A notable strength is the "channel-swap consistency" evaluation, which rigorously tests whether the model's spatial reasoning is grounded in binaural cues rather than language priors or monaural artifacts. The model demonstrates that spatial awareness can be added without degrading general audio understanding, maintaining competitiveness with state-of-the-art 8B-scale LALMs.
The paper provides high reproducibility by releasing source code and model checkpoints on GitHub and Hugging Face. Detailed implementation specifics, including data synthesis parameters, training curricula, and prompt templates for data generation, are provided in the appendices. The use of standard libraries like Pyroomacoustics and public datasets (LibriSpeech, AudioCaps) further enhances reproducibility.
The model relies heavily on synthetic data for spatial supervision, which may limit its performance on real-world spatial audio with complex, unmodeled acoustic phenomena (e.g., non-stationary noise, irregular room geometries). The spatial encoder is limited to binaural input, and the maximum number of overlapping sources in training is capped at 3, which may not cover all real-world scenarios. Additionally, the model's performance on fine-grained spatial details in complex scenes remains less effective compared to coarse-grained attributes.
This work bridges the gap between general audio understanding and spatial perception, offering a unified framework that can be applied to immersive audio applications, virtual reality, and assistive technologies for the visually impaired. The open-source release of the model and data pipeline facilitates further research in spatial audio-language modeling and encourages the development of more human-like auditory perception systems. MiDashengLM-Spatial unifies general audio understanding and spatial awareness in a single open-source model. The paper presents a rigorous integration of spatial and semantic encoders via hierarchical conditioning, supported by a large-scale synthetic data pipeline and comprehensive evaluations that demonstrate the model's ability to perform spatial reasoning without compromising general audio capabilities.
A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets are available at https://sepgen.github.io/
Primary: Tel Aviv University
All Institutions: Tel Aviv University, Google
SepGen introduces a unified framework for multi-stem audio-video generation and separation by extending a pretrained joint generator with a protected audio-mix channel and block-diagonal caption routing, enabling the extraction of individual source waveforms necessary for 4D spatial audio rendering. The technical contribution is robust, leveraging a two-stage training strategy and an "Estimated Separation" inference schedule to ensure stems remain faithful to their captions while maintaining the integrity of the original audio-mix, effectively solving the problem of inaccessible source separation in joint audio-video synthesis.
The paper proposes SepGen, a framework that extends a pretrained joint audio-video generator (specifically LTX-2.5) to output individual source stems alongside the video and mixed audio. The core technical contribution is a parameter-efficient adaptation using LoRA on the audio stream, where stems are treated as additional channels. Key innovations include a "protected" audio-mix channel that bypasses the adapter to preserve the backbone's original output quality, block-diagonal caption routing to ensure stems attend only to their specific captions, and a two-stage training strategy (separation first, then generation) that allows a single checkpoint to handle both tasks by varying the noise level of the audio-mix. The "Estimated Separation" inference schedule, where stems attend to the denoised estimate of the audio-mix rather than the noisy latent, is a clever mechanism to improve consistency during generation.
The evaluation covers both generation and separation modes. For separation, the model is compared against state-of-the-art language-queried separators (AudioSep, FlowSep, SAM Audio) on real recordings and cross-generator data. SepGen outperforms these baselines, particularly in speech separation, and maintains performance even when specific spoken lines are removed from captions. For generation, the paper evaluates the quality of the generated stems and their alignment with the video. The inclusion of a human A/B study adds qualitative depth. The use of a small dataset (4,843 examples) is a limitation, but the reliance on a strong pretrained backbone mitigates this.
The authors provide code, checkpoints, and datasets at the project URL. The detailed description of the training schedule, attention masks, and LoRA gating mechanisms in the method section suggests high reproducibility. The specific hyperparameters (e.g., threshold $\tau_g=0.97$, probability 0.3 for clean mix in Stage 2) are explicitly stated.
The model is currently limited to $N=2$ sources. The training dataset is relatively small (4,843 examples), which may limit generalization to complex scenes with many overlapping sources. The method relies heavily on the quality of the pretrained backbone (LTX-2.5); if the backbone's audio quality is poor, the stems will inherit these limitations. The paper does not extensively discuss computational overhead compared to the base model, though LoRA suggests it is minimal.
This work is significant for the field of 4D audio-visual scene rendering. By providing individual source waveforms, it enables spatial audio rendering from novel viewpoints, which is currently impossible with standard joint generators that only output a mixed soundtrack. It bridges the gap between generative video models and spatial audio applications, potentially impacting virtual reality, film production, and interactive media. SepGen introduces a unified framework for multi-stem audio-video generation and separation by extending a pretrained joint generator with a protected audio-mix channel and block-diagonal caption routing, enabling the extraction of individual source waveforms necessary for 4D spatial audio rendering. The technical contribution is robust, leveraging a two-stage training strategy and an "Estimated Separation" inference schedule to ensure stems remain faithful to their captions while maintaining the integrity of the original audio-mix, effectively solving the problem of inaccessible source separation in joint audio-video synthesis.
Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve $1.46\times$ and $1.59\times$ deployment-path wall-clock speedups and $1.61\times$ and $1.80\times$ Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.
Primary: University of Seoul
All Institutions: University of Seoul
[One sentence main contribution]. [The paper introduces DiffuPlex, a rolling masked diffusion framework that accelerates full-duplex spoken dialog models by predicting multiple future interaction frames in a single backbone pass, achieving significant computational speedups while preserving interaction quality through careful confidence-based retention and revision mechanisms.].
The paper proposes DiffuPlex, a "rolling masked diffusion" framework to accelerate full-duplex spoken dialog models. The core innovation is replacing the standard autoregressive step (one backbone forward pass per 80ms frame) with a mechanism that predicts a block of future user and assistant frames in a single "backbone wake." It uses learned MASK tokens for unavailable future inputs and trains via self-distillation from a frozen autoregressive backbone using LoRA adapters. The inference process involves predicting a future block, consuming a confident prefix of the assistant's speech, and checking user predictions against actual incoming audio. If a divergence is detected, it revises only the unplayed future. This approach is technically sound and addresses a real computational bottleneck in fine-grained full-duplex models. The distinction between "Listen" (consuming only silence predictions) and "Speak" (consuming speech predictions) policies is a practical and well-motivated design choice.
The experiments are comprehensive, covering full-duplex interaction quality (Full-Duplex-Bench), general spoken language capability (VoiceBench, OpenAudioBench), human evaluation (NMOS/DMOS), and inference efficiency (wall-clock and Core LM speedups). The results show significant speedups (1.46x-1.59x wall-clock) while largely preserving interaction quality. The analysis of why "Speak" mode degrades speech naturalness (loss of same-frame semantic conditioning for acoustic codebooks) is insightful and demonstrates a deep understanding of the underlying architecture. The comparison against deferred-execution baselines (skipping frames and emitting silence) effectively highlights the benefit of predicting the future rather than just skipping computation.
The paper provides detailed descriptions of the training layout, masking schedules, and inference policies. It specifies the backbone used (personaplex-rl-seamless) and the dataset (Seamless Interaction). While code release is planned post-review, the level of detail in the appendices (referenced but not fully provided in the text snippet, though implied to be extensive) suggests good reproducibility. The use of standard components (LoRA, Mimi codebooks) aids in understanding the implementation.
The method is currently adapted to a specific backbone (PersonaPlex family) and may not transfer automatically to other architectures, as noted in the limitations section. The "Speak" policy suffers from speech quality degradation due to architectural constraints (missing same-frame conditioning), which limits its practical utility compared to the "Listen" policy. The speedups, while significant, are moderate (around 1.5x), and the method adds complexity to the inference pipeline.
This work contributes to the efficiency of real-time speech interaction systems, which is crucial for deploying LLMs in conversational agents. By reducing the computational cost of full-duplex models, it makes such advanced interaction styles more feasible on edge devices or in high-concurrency server environments. The insights into the trade-offs between prediction confidence, revision, and speech quality are valuable for the broader community working on speech LLMs. [One sentence main contribution]. [The paper introduces DiffuPlex, a rolling masked diffusion framework that accelerates full-duplex spoken dialog models by predicting multiple future interaction frames in a single backbone pass, achieving significant computational speedups while preserving interaction quality through careful confidence-based retention and revision mechanisms.].