Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.
Primary: Brown University
All Institutions: Brown University, MIT, Yale University, The University of Hong Kong
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Sonic-AMG introduces a novel framework for streaming human motion generation from spatial audio, addressing key limitations of prior offline methods. The core methodology is built upon a causal transformer with a flow head, operating on continuous motion latents from a pretrained VAE. A significant methodological contribution is the separation of acoustic content ("what") and body-relative source geometry ("where") into distinct conditioning tokens. This design choice, supported by ablations, allows the model to better disentangle and utilize these modalities. The 11-dimensional spatial descriptor, inspired by perceptual research, is a thoughtful design for capturing relevant spatial cues like proximity, relative motion, and looming. The paper effectively tackles two critical training-inference mismatches inherent in autoregressive streaming generation. First, the "history mismatch" (teacher-forced training vs. generated histories at inference) is addressed by "rehearsal training." This innovative approach combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. This hybrid strategy allows the model to learn stable long-range dependencies while also gaining robustness to accumulated errors from its own predictions. Second, the "spatial conditioning mismatch" (ground-truth SSL vs. generated root poses) is resolved by "spatial feedback." This mechanism recomputes body-relative source locations from generated root poses during both rehearsal training and inference, ensuring consistency between the generated motion and the spatial conditions. The integration of these components within a causal transformer and flow-matching framework is technically sound and well-justified. The use of classifier-free guidance during inference further enhances generation quality.
The experimental evaluation is comprehensive and well-structured, primarily using the SAM dataset. The paper establishes state-of-the-art performance for Sonic-AMG on FID and R-precision metrics, demonstrating both distributional fidelity and conditioning alignment. A crucial aspect of the evaluation is the distinction between "GT SSL" (ground-truth spatial conditioning, for fair comparison with baselines) and "Online SSL" (spatial feedback from generated motion, reflecting real-world streaming). Sonic-AMG maintains strong performance under the more challenging Online protocol, which is a significant achievement. The ablation studies are particularly strong, providing clear evidence for the efficacy of the proposed components: 1. **Rehearsal training vs. two-forward training:** Rehearsal training consistently outperforms various two-forward schedules, validating the benefit of recursive history construction. 2. **Role of teacher-forced supervision:** Demonstrates that combining self-rollouts with full-sequence supervision is crucial for fidelity and alignment. 3. **Effect of spatial feedback:** Shows that training with spatial feedback dramatically improves performance under the Online protocol, confirming its necessity for consistent spatial conditioning. 4. **Separate spatial tokens:** Ablation against a "Fused" approach confirms the advantage of distinct acoustic and spatial tokens. 5. **Learned acoustic features:** Shows that simpler, learned log-mel features outperform MOSPA's high-dimensional audio descriptors, supporting the hypothesis that separating "what" from "where" helps the acoustic branch focus on content. Beyond quantitative metrics, the paper provides compelling qualitative results, including examples of recognizable and spatially appropriate reactions, and the ability to generate coherent motion across complete audio clips and successive sound events. The demonstration of real-time streaming efficiency, with significantly reduced initial response latency compared to MOSPA, is a strong practical contribution. The inclusion of physical simulation retargeting to a Unitree G1 robot further showcases the robustness and applicability of the generated motions. The "No-SSL" variant, which derives spatial tokens directly from binaural audio, is an excellent exploration of alternative inference scenarios, demonstrating the model's flexibility and potential for deployment without explicit source location inputs.
The paper includes a dedicated "Reproducibility statement" and detailed appendices covering implementation details, training protocols, and runtime measurements. This includes specifics on architecture (transformer layers, heads, width, flow head details), optimization (AdamW, learning rate, batch size), training hardware and time, motion autoencoder pretraining, acoustic and spatial encoding, guidance, and training sequence construction for both teacher-forced and rehearsal training. The authors commit to releasing code and models upon acceptance, which, combined with the extensive details provided, suggests high reproducibility.
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Primary: FAIR, Meta Superintelligence Labs
All Institutions: The University of Texas at Austin, FAIR, Meta Superintelligence Labs
The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
The paper proposes RUBATO, a multistream positional encoding mechanism that assigns independent "clocks" to different data streams (real-time audio vs. internal thinking) in full-duplex speech models. The core innovation is decoupling the positional indices of asynchronous streams to prevent temporal distortion when the decoding rate of the thinking stream varies. The authors derive three variants: TM-RoPE (subspace partitioning), Key-Clock (conditional rotation based on key token clock), and Full-Clock (rotating all dimensions by the key's clock). The methodology is mathematically sound, leveraging the properties of Rotary Position Embeddings (RoPE) to allow zero-shot interpolation of inference speeds. The implementation strategy, which expands hidden dimensions during training to fit standard FlashAttention kernels and uses separate KV caches during inference, is a clever engineering solution that avoids custom Triton kernels, enhancing practicality.
The experiments are rigorous, evaluating the model on Spoken GSM8K and BIG-Bench Audio, including a novel "Overlapped" scenario where the model must handle interruptions while reasoning. The results demonstrate that RUBATO-Full maintains high accuracy (48-51%) across varying decoding budgets, whereas baselines collapse to near-zero accuracy at interpolated speeds. The paper provides strong evidence for the utility of the method in reducing time-to-answer by leveraging faster hardware without retraining. The inclusion of UTMOS and Speaker Similarity metrics for filler audio quality adds depth to the evaluation, confirming that the architectural changes do not degrade acoustic quality.
The paper provides a detailed reproducibility statement, including closed-form equations for the attention mechanisms and clear descriptions of the training data synthesis process. The use of standard backbones (Gemma-4) and tokenizers (NanoCodec) aids reproducibility. However, the specific hyperparameters for the training mixture and the exact construction of the synthetic datasets are deferred to the appendix, which is not fully visible in the provided text, though the main text describes the logic clearly.
The primary limitation is the reliance on synthetic data for training, as no suitable full-duplex spoken reasoning corpora exist. The evaluation is limited to mathematical and logical reasoning tasks (GSM8K, BIG-Bench), which may not generalize to all conversational domains. Additionally, the method assumes a specific interleaving strategy (SCoT), and its applicability to other full-duplex architectures (e.g., concatenation-based) is not explored. The "Full-Clock" variant, while best performing, requires duplicating the head dimension, which increases memory usage.
This work has significant implications for the deployment of real-time AI agents. By decoupling reasoning speed from real-time constraints, it enables more efficient use of heterogeneous hardware. The concept of independent clocks for asynchronous streams is a generalizable architectural pattern that could apply to other multimodal or multi-agent systems requiring variable-rate processing. It addresses a critical bottleneck in making LLMs truly interactive and responsive in speech-based interfaces. The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
Primary: Meta Superintelligence Labs
All Institutions: Meta Superintelligence Labs, University of Southern California
The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
The paper proposes ATTEST, a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) specifically tailored for long-audio temporal grounding. The core novelty lies in the design of three distinct reward signals: (1) a Counterfactual Audio-Grounding (CAG) reward that penalizes predictions surviving the removal of acoustic evidence (via silence or corruption), effectively acting as a hallucination penalty; (2) an Attention Density Grounding reward that encourages the model to attend to the specific audio tokens corresponding to the time spans it claims in its output; and (3) a token-wise fine-grained advantage mechanism that assigns credit at the segment level rather than the sequence level, which is crucial for long, dense captions. This approach avoids modifying the base model's architecture or vocabulary, making it a plug-and-play improvement for existing Large Audio Language Models (LALMs). The methodology is sound, addressing the specific failure modes of LALMs on long audio (hallucination and temporal drift) with targeted RL interventions.
The authors introduce EgoSCAPE, a new benchmark derived from Ego4D with 700 manually verified audio clips up to 5 minutes long, featuring dense captions, sound sources, and temporal grounding queries. This is a significant contribution as existing benchmarks often lack dense annotations for long-form audio. Experiments show that ATTEST outperforms strong baselines (including Gemini 3.1 Pro, GPT-4o Audio, and specialized models like TimeAudio and AudioFlamingo 3) on both the new EgoSCAPE benchmark and existing datasets (Clotho-Moment, CASTELLA). The ablation studies convincingly demonstrate the individual contribution of each reward component, particularly the CAG reward in reducing hallucinations. The results on general audio understanding benchmarks (MMAR, LongAudioBench) confirm that the method does not cause catastrophic forgetting, which is a common issue with temporal grounding fine-tuning.
The paper provides a detailed description of the benchmark creation pipeline, including the use of BEATs for initial detection and LLMs for verification, followed by human verification. The RL training details (LoRA, GRPO parameters, reward weights) are specified. However, the code and the EgoSCAPE benchmark itself are not explicitly linked in the provided text (though a reproducibility statement is made). The reliance on specific proprietary models (Gemini, GPT) for baselines limits full reproducibility of those comparisons, but the open-source base models (Qwen, AudioFlamingo) allow for partial reproduction.
The benchmark is limited to 5-minute clips, which is still shorter than some real-world long-form audio scenarios (e.g., hours-long podcasts). The method relies on the availability of attention weights, which may not be accessible in all proprietary or optimized inference engines. The counterfactual reward requires generating a reference model output on corrupted audio, adding computational overhead during training. The paper acknowledges that the method is currently evaluated on audio-only tasks and does not extend to video or omni-modal temporal grounding in the main experiments.
This work addresses a critical bottleneck in deploying LALMs for real-world applications where audio is long and unstructured. By providing a rigorous benchmark (EgoSCAPE) and a training framework (ATTEST) that reduces hallucinations without architectural changes, it enables more reliable temporal reasoning in audio AI. This has implications for accessibility tools, video indexing, and autonomous agents that need to understand and locate specific events in long audio streams. The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.
Primary: Tsinghua University
All Institutions: Tsinghua University, ByteDance
ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
The paper proposes ELLSA 2, a unified end-to-end framework that integrates full-duplex speech interaction with streaming gesture and facial expression generation. The core methodological contribution is the "contextual semantic call" mechanism, which decouples the decision of *whether* to perform a semantic gesture from *how* to realize it. This is achieved via a binary classification head on the LLM backbone's hidden states, which triggers the retrieval of a curated exemplar template to guide the autoregressive gesture generator. The architecture uses a chunk-wise autoregressive synthesizer that interleaves speech, listening, and motion tokens, allowing non-verbal behaviors to persist during verbal silence (listening states). The use of RVQ-VAE for motion tokenization and a curriculum training strategy to integrate modalities is technically sound and well-motivated.
The experimental setup is robust, introducing a new human-verified benchmark (138 clips) specifically for dialog-aware gesture generation, covering full-duplex behaviors and semantic gestures. The evaluation employs both objective metrics (prosodic alignment, activity ratio) and subjective human evaluation, along with MLLM-as-a-judge metrics. The results demonstrate that ELLSA 2 outperforms cascaded baselines and remains competitive with state-of-the-art full-duplex speech models. The semantic template guidance shows a significant improvement in semantic gesture quality (human score 2.014 to 2.711). The correlation analysis between MLLM judges and human annotators (0.853/0.919) provides strong evidence for the utility of the proposed automatic metrics.
The paper provides extensive implementation details, including tokenizer specifications, training hyperparameters, and data curation pipelines. The authors commit to releasing code, data, and model checkpoints upon acceptance. The detailed description of the semantic exemplar library construction and the evaluation protocols enhances reproducibility.
The system relies on a curated library of 3,227 exemplars for semantic gestures, which may limit the diversity of semantic actions compared to fully generative approaches. The evaluation, while improved, is still based on a relatively small test set (138 clips). The model's performance on long-horizon coherence of gestures over extended conversations is not deeply analyzed.
This work advances the field of human-computer interaction by moving beyond text or speech-only interfaces to fully embodied, multimodal agents. The proposed evaluation metrics and benchmark will be valuable for the community working on co-speech gesture generation and full-duplex dialogue systems. The integration of semantic understanding with physical gesture generation has implications for robotics, virtual avatars, and immersive communication. ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.
Primary: University of Melbourne
All Institutions: University of Melbourne, National Computational Infrastructure (NCI Australia)
VoxTurn introduces a 0.8B speech LLM that answers multiple turn-taking questions in a single forward pass using a block-isolating attention mask, achieving commercial-competitive performance on end-of-turn detection with significantly improved efficiency and multilingual robustness.
The paper proposes VoxTurn, a 0.8B speech LLM based on Qwen3-ASR-0.6B, adapted for turn-taking decisions. The core methodological innovation is the "one-pass multi-question" inference mechanism using a block-isolating attention mask. This allows the model to answer multiple related questions (e.g., end-of-turn, interruption, backchannel, invalid input) in a single forward pass, mathematically equivalent to asking them sequentially but significantly more efficient. The training strategy is also notable: instead of using segmented utterances with static labels, the authors train on real conversational pauses, sampling decision times uniformly from within the pause and deriving labels from the timing of subsequent speech. This addresses the distribution mismatch between training (segmented) and deployment (streaming/real-time) scenarios. The use of a linear head on hidden states for multiple-choice scoring rather than text generation is a standard but effective adaptation for classification tasks in LLMs.
The experimental evaluation is comprehensive and rigorous. The authors evaluate on eot-bench (14 languages) and TurnBench (two-party conversations). Key results include 11.5% false cut-offs in English at 300ms latency, comparable to commercial detectors, and the best open-model performance on the 14-language mean (19.9%). The ablation studies clearly demonstrate the impact of real pause data versus segmented data, showing that the data strategy is the primary driver of performance gains. The efficiency analysis shows a 3.7x speedup for multi-question inference compared to separate passes. The integration tests with Pipecat and LiveKit frameworks provide strong evidence of practical deployability, showing reduced cut-off rates and lower latency in real-world agent setups.
The paper provides high reproducibility. The authors release the model weights, code, and framework plugins on Hugging Face. They provide detailed training hyperparameters, data mixture descriptions, and the specific attention mask implementation details. The use of open-source benchmarks (eot-bench, TurnBench) and the release of an interactive demo further enhance reproducibility and verification.
The model is evaluated primarily on English and a set of 14 languages, but the performance gap with commercial detectors in some languages (Hindi, Korean) remains. The reliance on a specific backbone (Qwen3-ASR) may limit generalizability to other architectures. The "zero-shot" performance on TurnBench is strong, but the model was trained on data that overlaps conceptually with the task, so true generalization to unseen conversational dynamics is hard to assess fully. The latency budget of 300ms is tight, and performance may degrade if hardware constraints are stricter.
This work has significant impact on the development of real-time voice agents. By providing an open-source model that matches commercial performance in turn-taking, it lowers the barrier to entry for building high-quality conversational AI. The multi-question approach is a valuable architectural pattern that can be applied to other sequential decision-making tasks in speech. The focus on real pause data sets a new standard for training data curation in this domain. VoxTurn introduces a 0.8B speech LLM that answers multiple turn-taking questions in a single forward pass using a block-isolating attention mask, achieving commercial-competitive performance on end-of-turn detection with significantly improved efficiency and multilingual robustness.
Large audio language models (LALMs) have made substantial progress in audio understanding. However, in high-throughput applications requiring frequent selection, judgment, and scoring, autoregressive text generation introduces additional computational overhead and response latency. To address this limitation, we introduce Qwen-Audio-Jev by augmenting Qwen-Audio-3.1-ASR-Flash-Next with a dedicated Jev decision head. The model unifies selection, binary judgment, and scoring through three decision primitives: CHOICE, NOUL, and SCORE. Given audio, a natural-language question, and an explicit answer specification, it directly produces a task-conditioned probability distribution or a numerical score through typed decision readouts. Evaluation across 18 labeled configurations covers all three primitives: for CHOICE, Qwen-Audio-Jev achieves accuracies of 78.70% on MMAU-mini and 84.06% on MMSU; for NOUL, it achieves an equal error rate of 6.50% on the VoxCeleb1-O speaker verification benchmark; and for SCORE, the model achieves a normalized mean absolute error of 0.1062 on NISQA overall quality assessment, with Pearson and Spearman correlations of 0.8210 and 0.8074, respectively. Compared with the original Qwen-Audio-3.1-ASR-Flash-Next, Qwen-Audio-Jev achieves an approximately 20x inference speedup. It demonstrates its potential for efficient, high-throughput audio decision-making in applications such as large-scale audio screening, speaker verification, and automated quality assessment.
Primary: Alibaba Group
All Institutions: Alibaba Group, Token Foundry
The paper introduces a unified decision head for audio LLMs that replaces autoregressive decoding with direct typed readouts for selection, judgment, and scoring, achieving significant inference speedups. While the architectural contribution is incremental, the integration of calibration-focused reinforcement learning and the comprehensive evaluation across heterogeneous audio tasks demonstrate a practical path toward efficient, high-throughput audio understanding systems.
The paper proposes a "Jev decision head" attached to the Qwen-Audio-3.1-ASR-Flash-Next backbone to replace autoregressive text generation with direct probability/score readouts for three primitives: Choice (categorical), Noul (binary), and Score (ordinal). The methodology is sound in its architectural simplicity, leveraging the existing MoE decoder's hidden states. The introduction of a Reinforcement Learning for Calibrated Decisions (RLCD) stage using Brier and ranked-probability scores is a technically interesting addition aimed at improving probability calibration, which is often neglected in standard fine-tuning. However, the core idea of adding classification heads to LLMs for specific tasks is not new, and the "Jev" terminology appears to be a proprietary or niche framework that limits the generalizability of the contribution. The training strategy of jointly optimizing heterogeneous tasks with specific loss functions (CE for Choice, BCE for Noul, and a composite loss for Score) is standard but well-executed.
The evaluation covers 18 configurations across 8 Choice, 4 Noul, and 6 Score tasks. The results are competitive, achieving 84.06% on MMSU and 6.50% EER on VoxCeleb1-O. The claim of a 20x inference speedup is significant for high-throughput applications. However, the paper lacks comparisons against other non-autoregressive audio models or specialized classifiers (e.g., ResNet-based for speaker verification, or dedicated MOS predictors like UTMOS/DNSMOS). The comparison is primarily against the autoregressive version of the same backbone, which limits the context of the performance gains. The Score results on NISQA show good correlation (0.82 Pearson) but higher MAE compared to state-of-the-art specialized MOS predictors, suggesting the unified approach may sacrifice some peak accuracy for versatility.
The paper provides architectural details (40 layers, 2048 hidden dim, 256 experts) and loss formulations. However, it does not provide code, model weights, or detailed hyperparameter settings for the RLCD stage (e.g., learning rates, number of steps, exploration temperature). The reliance on the proprietary "Qwen-Audio-3.1-ASR-Flash-Next" backbone further limits reproducibility for external researchers who may not have access to this specific checkpoint.
1. Lack of comparison with specialized non-LLM baselines for specific tasks (e.g., speaker verification, MOS prediction). 2. The "Jev" framework is not widely known, making the contribution harder to situate in the broader literature. 3. The RLCD stage is described theoretically but lacks ablation studies showing its specific contribution to calibration compared to SFT alone. 4. No streaming or real-time latency analysis beyond the theoretical speedup claim. 5. The model is evaluated on a relatively small set of benchmarks, and the "18 configurations" are not fully detailed in the provided text (tables are referenced but not fully visible in the prompt's text snippet, though the abstract summarizes key metrics).
The work addresses a practical bottleneck in deploying LALMs for high-volume tasks like screening and verification. By decoupling the decision output from text generation, it enables faster, more deterministic, and easier-to-integrate pipelines. This could facilitate the deployment of audio AI in edge devices or large-scale cloud services where latency and cost are critical. The focus on calibrated probabilities (via RLCD) is particularly valuable for safety-critical applications where uncertainty estimation is required. The paper introduces a unified decision head for audio LLMs that replaces autoregressive decoding with direct typed readouts for selection, judgment, and scoring, achieving significant inference speedups. While the architectural contribution is incremental, the integration of calibration-focused reinforcement learning and the comprehensive evaluation across heterogeneous audio tasks demonstrate a practical path toward efficient, high-throughput audio understanding systems.
Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.
Primary: Unknown (Likely Alibaba/Tongyi Lab based on Qwen3/MOSS dependencies)
All Institutions: Unknown
Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
The paper proposes Soloist, a generative model for blind voice separation that frames the task as autoregressive generation of audio-codec sequences. The core methodological contribution is a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT) on synthetic mixtures with isolated targets and, crucially, on real recordings with only speaker timeline annotations (who speaks when), which helps the model learn count inference and activity patterns without waveform targets; and (2) Reinforcement Learning (RL) on real recordings using a "model-judge" reward. The RL component is particularly novel, employing a multi-step verification process where an audio-language model (Qwen3-Omni) describes candidates independently, and a text model (Qwen3.6) ranks them based on these descriptions, with a consistency check (Kendall's tau) under shuffled/reversed orders to filter out order-biased or unstable feedback. This addresses the lack of ground-truth isolated sources in real-world data. The architecture leverages a Qwen3-4B backbone with a local transformer for codec code prediction, integrating continuous and discrete audio features.
The evaluation is comprehensive, covering overlapping speech (LibriMix, WildTV), noisy environments (ViW), and singing vocals (SongYT, MIR-1K, MUSDB18-HQ). The paper reports strong results in PI-WER and DNSMOS, outperforming traditional separators (SepFormer, MossFormer) and query-based extractors. The inclusion of a full-duplex conversation pipeline demonstrates practical utility, showing how the model can be integrated into a larger system for continuous speaker track assembly. The judge validation section provides evidence that the RL reward signal correlates with objective quality metrics (PI-WER) on known-source mixtures, which is a critical sanity check for the RL approach.
The paper provides detailed descriptions of the data construction, training objectives, and RL hyperparameters. However, the reliance on specific, potentially proprietary or large-scale models (Qwen3-Omni, MOSS-Audio-Tokenizer, MOSS-Transcribe-Diarize) and the complexity of the judge pipeline make exact reproduction difficult without access to these specific components and the in-house datasets (e.g., the "in-house song recordings" and specific Pixabay subsets). The code is not explicitly linked in the provided text, though the use of open-source base models (Qwen3) suggests a path to partial reproduction.
The method is computationally expensive due to the autoregressive generation of codec codes and the multi-step RL judge process. The "blind" separation relies on the model's ability to infer the correct number of speakers, which can fail in highly overlapping or noisy scenarios. The RL reward, while robust to order bias, is still based on an LLM's judgment of audio descriptions, which may not perfectly align with human perceptual quality or subtle acoustic artifacts. The singing separation uses teacher-generated pseudo-labels, which limits the fidelity to the true studio vocal.
This work bridges the gap between traditional signal processing for speech separation and modern generative AI techniques. By using LLMs as judges for RL in audio tasks, it offers a template for training audio models on unpaired, real-world data. The application to full-duplex conversation processing highlights its potential for real-time or near-real-time conversational AI systems that require robust handling of overlapping speech and noise. Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.
Primary: Institute of Acoustics, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences
The paper presents a novel and effective framework for noise-aware Lombard speech synthesis by integrating FiLM conditioning with Flow-GRPO reinforcement learning. It demonstrates that directly optimizing for intelligibility via RL yields superior performance compared to supervised methods or static style adjustments, providing a robust solution for real-world voice agent deployment.
The paper proposes NIL-TTS, a framework that integrates Feature-wise Linear Modulation (FiLM) into a flow-matching TTS backbone (F5-TTS) to condition speech generation on ambient noise. The core methodological contribution is a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT) on paired Lombard corpora to establish baseline acoustic mappings, and (2) Flow-GRPO post-training to optimize for intelligibility. The use of Flow-GRPO is significant as it adapts Group Relative Policy Optimization to continuous flow-matching models by reformulating the deterministic ODE into a stochastic SDE for exploration. The reward function combines ASR-derived intelligibility (WER) and speaker similarity, directly targeting the goal of the Lombard effect rather than just acoustic mimicry. The architectural choice to apply FiLM only to deeper transformer blocks is a thoughtful design decision to preserve phonetic alignment.
The experimental evaluation is rigorous and comprehensive. The authors utilize the Seed-TTS benchmark for objective evaluation, testing across multiple noise types (Babble, Factory, Pink, White) and SNR levels. They provide strong baselines, including vanilla F5-TTS, signal-processing enhancements (SS, SSDRC), and other neural Lombard methods (MetaVoice-1B, F5-TTS-Style). The inclusion of an "RL-only" baseline effectively isolates the contribution of the environmental conditioning mechanism from the RL post-training itself. Subjective evaluation follows ITU-T P.807 standards, confirming perceptual gains. Acoustic parameter analysis (F0, Formants, Spectral Tilt) provides mechanistic insight into how the model achieves intelligibility, showing that the RL stage rebalances rather than just amplifies Lombard cues.
The paper provides detailed training configurations, dataset descriptions, and hyperparameters (e.g., learning rates, NFE, diffusion coefficients). The use of public datasets (EMALG, LombardGrid, WenetSpeech4TTS, DNS Challenge) and standard models (F5-TTS, Whisper, Paraformer) enhances reproducibility. However, the specific implementation of the Flow-GRPO adaptation to the F5-TTS architecture might require careful engineering to replicate, though the mathematical formulation is clearly provided.
The primary limitation is the reliance on ASR models for the reward signal, which may not perfectly correlate with human perceptual intelligibility in all edge cases. The computational cost of the RL post-training phase, while not explicitly detailed in terms of total GPU hours, is likely significant given the SDE rollouts. Additionally, the model is trained primarily on Mandarin data for the RL stage, and while cross-lingual transfer to English is observed, the generalization to other languages or noise types not seen in training is not extensively tested.
This work has significant implications for the deployment of voice-based conversational agents in real-world, noisy environments. By enabling TTS systems to autonomously adapt to acoustic conditions, it moves the field towards more robust and human-like interaction. The integration of RL for direct optimization of downstream metrics (intelligibility) rather than just reconstruction loss is a promising direction for generative audio models. The paper presents a novel and effective framework for noise-aware Lombard speech synthesis by integrating FiLM conditioning with Flow-GRPO reinforcement learning. It demonstrates that directly optimizing for intelligibility via RL yields superior performance compared to supervised methods or static style adjustments, providing a robust solution for real-world voice agent deployment.
Full-duplex spoken language models decide at every frame whether to take the floor, defining a conversational personality -- yet each set of weights encodes exactly one, which human speakers bracket rather than share. Existing controls mostly act from outside; none identifies the quantity that sets how much the model speaks. We identify it: a single late-layer residual-stream direction, the initiative vector, extracted from the model's own conversations without training or annotation. The same direction reads the decision, anticipating onsets up to 800 ms ahead, and shifts it: scaled during generation, it moves speech ratio from 6-12% to 42-61% (no steering: 21-30%), consistently across conversations, weights, and corpora. A short reference recording suffices to transfer a speaker's turn-taking style to the model, and the transfer is reference-specific: swapping in another speaker's reference worsens the match in 93-98% of cases, whereas the best uniform dose is indistinguishable from no steering. Turn-taking style is not frozen into the weights but a direction that can be read, shifted, and calibrated per person. Audio examples are available at https://elpsykongloo.github.io/initiative-vector/.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, The University of Melbourne, Lulula
The paper identifies a single residual-stream direction, the "initiative vector," that linearly encodes and controls turn-taking behavior in full-duplex spoken language models. By demonstrating that this vector can be extracted without training and used to steer speech ratio and transfer individual turn-taking styles from short reference recordings, the work provides a novel, efficient, and interpretable mechanism for personalizing conversational AI, bridging the gap between fixed model weights and dynamic human-like interaction styles.
The paper proposes a method to identify and manipulate a specific "initiative vector" in the residual stream of full-duplex spoken language models (specifically the Moshi architecture). The methodology involves extracting this vector via class-mean contrasts of frames preceding speech onsets versus silent frames, requiring no additional training or annotation. The core technical contribution is the demonstration that this single direction linearly encodes the model's propensity to speak and can be scaled during generation to control turn-taking behavior. The authors further develop a reference-guided transfer mechanism that calibrates this steering dose based on a short reference recording of a target speaker's style, using descriptors like speech ratio and overlap onsets. The approach is elegant in its simplicity, leveraging linear interpretability techniques (similar to activation steering in LLMs) applied to the temporal dynamics of speech models.
The experiments are conducted on three distinct full-duplex weights (Moshika, Moshika-RL, PersonaPlex) using the Seamless Interaction and CANDOR corpora. The evaluation metrics include AUC for onset prediction, speech ratio changes, and distance metrics for style transfer. The results show that the initiative vector anticipates onsets with high AUC (up to 0.968) and that scaling it monotonically increases speech ratio. The style transfer experiments demonstrate that using a specific speaker's reference yields significantly better matches than uniform doses or other speakers' references. The statistical analysis includes confidence intervals and comparisons against baselines, providing robust evidence for the claims.
The paper provides sufficient detail for reproducibility, including the specific layer (L30), the extraction method (class means), the dose scaling factor, and the descriptors used for reference matching. The use of public datasets (Seamless Interaction, CANDOR) and open-source models (Moshi) enhances reproducibility. However, the specific implementation details of the steering mechanism during generation (how exactly the residual is modified in the KV cache) could be more explicitly detailed in the main text, though the equations provided are clear.
The study is limited to the Moshi architecture and its variants; generalizability to other full-duplex architectures (e.g., those with different temporal resolutions or attention mechanisms) is not tested. The steering is applied uniformly across all frames, which may not capture more nuanced, context-dependent turn-taking strategies. Additionally, the "style" is defined by a few acoustic descriptors, which may not fully capture the complexity of human conversational personality.
This work has significant implications for the development of more natural and personalized spoken dialogue systems. By enabling fine-grained control over turn-taking without retraining, it offers a practical path to adapting models to different user preferences or application contexts (e.g., tutoring vs. companionship). It also contributes to the interpretability of speech models, providing insight into how conversational decisions are computed within the network. The paper identifies a single residual-stream direction, the "initiative vector," that linearly encodes and controls turn-taking behavior in full-duplex spoken language models. By demonstrating that this vector can be extracted without training and used to steer speech ratio and transfer individual turn-taking styles from short reference recordings, the work provides a novel, efficient, and interpretable mechanism for personalizing conversational AI, bridging the gap between fixed model weights and dynamic human-like interaction styles.
We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.
Primary: Fish Audio
All Institutions: Fish Audio
Fish Audio Drama 3 introduces a 35B MoE speech model with a novel codebook-MTP decoder and preference-based RL, achieving state-of-the-art results in multilingual cloning and expressive performance while improving inference efficiency through architectural and engineering optimizations.
The paper introduces Fish Audio Drama 3, a 35B-parameter Mixture-of-Experts (MoE) speech generation model. The core architectural innovation is the replacement of the previous "Fast AR" module with a "Codebook Multi-Token-Prediction (MTP) Decoder." This decoder uses dedicated layers for each residual codebook that attend causally over the entire sequence, allowing fine acoustic details from reference audio to directly influence codebook predictions, rather than being compressed into a single backbone state. The backbone utilizes a hybrid of Gated DeltaNet (linear attention) and full attention layers to handle long contexts efficiently. A significant methodological contribution is the training pipeline, which employs a single learned reward model for Reinforcement Learning (RL) instead of rule-based rewards (like WER or speaker similarity). This addresses the conflict where expressive performance (whispering, laughing) often penalizes standard intelligibility/similarity metrics. The RL uses Group Relative Policy Optimization (GRPO) with large batches and no KL penalty, optimizing a preference model trained on human judgments.
The evaluation is comprehensive, covering multilingual cloning (MiniMax Multilingual Testset), editing (Ming-Freeform-Audio-Edit), and revision (Step-Audio-Edit). Drama 3 demonstrates state-of-the-art or competitive performance, particularly in multilingual cloning where it outperforms OmniVoice and FireRedTTS3-Base in multiple languages. The paper provides a strong ablation on the reward model, showing that the preference-based reward correctly prioritizes styled speech (e.g., whispering) over neutral speech, whereas rule-based rewards would penalize it. Inference efficiency is also rigorously evaluated, showing that despite having 7x more parameters than the previous model (S2), the new architecture and FP8 quantization allow for faster decoding (271 fps vs 177 fps on H200).
High. The model weights are released on Hugging Face and GitHub. The paper provides detailed descriptions of the architecture, training stages, and inference engine optimizations. While the specific data curation details are proprietary, the architectural and algorithmic details are sufficient for reproduction of the model structure. The use of open-source benchmarks and public baselines enhances reproducibility of the evaluation.
The model is large (35B parameters), which may limit deployment on edge devices despite the MoE efficiency. The paper relies on a proprietary reward model for RL, which is not released, making it difficult to fully reproduce the RL alignment process. The evaluation of "directed performance" is somewhat subjective and relies on the specific benchmarks provided, which may not cover all nuances of human direction.
This work significantly advances the field of expressive speech synthesis by unifying cloning, direction, editing, and multi-speaker scenes in a single model. The shift from rule-based to preference-based RL rewards is a crucial step towards more natural and controllable audio generation. The open-source release of such a large, high-performing model lowers the barrier for research and application in interactive characters, dubbing, and audiobooks. Fish Audio Drama 3 introduces a 35B MoE speech model with a novel codebook-MTP decoder and preference-based RL, achieving state-of-the-art results in multilingual cloning and expressive performance while improving inference efficiency through architectural and engineering optimizations.
Deep learning models of speech seldom represent the source-filter process by which speech is produced, relying instead on latent features with no physical correspondence to articulation or phonation. We propose SoFilNet, a self-supervised, differentiable Source-Filter Network that estimates the parameters of speech production directly from the raw waveform: a source, specified by fundamental frequency (F0), voicing and the aperiodicity of five frequency bands, and a filter, specified by ordered formant poles, additional poles that shape the rest of the spectral envelope, and antiresonance zeros. The model is trained only to reconstruct the input speech; no formant or pitch annotation is used. Since per-bin spectral losses favor blurred spectra wherever the model is uncertain, we train F0 and voicing against autocorrelation pseudo-labels refined with the model's own filter, aperiodicity against the periodic fraction of each band, and formant sharpness against spectral contrast. On VTR-FORMANTS, the estimated formants attain the lowest mean error among the reported methods, including supervised trackers, and on the unseen PTDB-TUG corpus the F0 frame error matches the best classical tracker and is lower than that of all learned trackers. The parameters are also consistent with speech production theory: aperiodicity rises with frequency and orders phone classes by the turbulence of their excitation, and the zeros deepen on nasal consonants, the lower one rising with place of articulation.
Primary: Indian Institute of Technology Hyderabad
All Institutions: Indian Institute of Technology Hyderabad
The paper presents a novel self-supervised differentiable source-filter model that accurately estimates speech production parameters from raw waveforms. By addressing the limitations of per-bin spectral losses with targeted auxiliary losses and a multi-resolution architecture, SoFilNet achieves state-of-the-art formant and pitch tracking accuracy while providing physically interpretable parameters consistent with speech production theory.
The paper proposes SoFilNet, a self-supervised differentiable source-filter model that estimates speech production parameters (F0, voicing, band aperiodicity, formants, zeros) directly from raw waveforms. The methodology is technically sophisticated, addressing the "blur" problem of per-bin spectral losses by introducing specific auxiliary losses: autocorrelation-based pseudo-labels for F0/voicing (refined via inverse filtering), periodic fraction loss for aperiodicity, and spectral contrast loss for formant sharpness. The use of a multi-resolution encoder with separate gates for source and filter parameters is a well-motivated architectural choice that aligns with the different temporal scales of glottal and vocal tract dynamics. The stick-breaking recursion for formant ordering is a clever constraint to ensure physical consistency without post-hoc tracking.
The evaluation is rigorous and comprehensive. The model is tested on VTR-FORMANTS for formant tracking, achieving the lowest mean error among reported methods, including supervised baselines. Pitch tracking is evaluated on the unseen PTDB-TUG corpus, where SoFilNet matches the best classical tracker (RAPT) and outperforms learned trackers. The interpretability analysis provides strong evidence that the learned parameters align with speech production theory (e.g., aperiodicity rising with frequency, zeros deepening on nasals). The ablation studies effectively demonstrate the contribution of zeros, the multi-resolution encoder, and the specific loss functions. The reconstruction quality (PESQ/STOI) from parameters alone further validates the information content of the estimated features.
The paper provides detailed descriptions of the architecture, loss functions, and training procedure. Hyperparameters such as learning rate, batch size, and number of epochs are specified. The use of standard datasets (TIMIT, VTR-FORMANTS, PTDB-TUG) and well-known baselines facilitates reproduction. However, the specific implementation details of the differentiable synthesizer and the exact code for the pseudo-label refinement process are not provided in the text, though a demo link is available.
The model is trained on TIMIT, which is a relatively small and dated dataset, potentially limiting generalization to diverse accents or noisy environments. The evaluation of formants is limited to F1-F3 (F4 excluded due to lack of hand-correction in the benchmark), which may not capture the full vocal tract response. The pitch evaluation, while strong, is on a single corpus (PTDB-TUG), and the model's performance on highly noisy or pathological speech is not assessed. The computational cost of the multi-resolution encoder and the iterative pseudo-label refinement may be higher than simpler baselines.
This work bridges the gap between data-driven deep learning models and physical speech production theory. By providing interpretable parameters that correspond to articulatory and phonatory features, SoFilNet can be used for speech analysis, synthesis, and modification tasks where physical interpretability is crucial. The self-supervised approach reduces the need for expensive expert annotations, making it scalable to larger datasets. The insights into the behavior of aperiodicity and zeros could inform future speech coding and synthesis models. The paper presents a novel self-supervised differentiable source-filter model that accurately estimates speech production parameters from raw waveforms. By addressing the limitations of per-bin spectral losses with targeted auxiliary losses and a multi-resolution architecture, SoFilNet achieves state-of-the-art formant and pitch tracking accuracy while providing physically interpretable parameters consistent with speech production theory.
Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
Primary: Meta Superintelligence Labs
All Institutions: Meta Superintelligence Labs, University of Southern California
The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
The paper proposes ATTEST, a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) specifically tailored for long-audio temporal grounding. The core novelty lies in the design of three distinct reward signals: (1) a Counterfactual Audio-Grounding (CAG) reward that penalizes predictions surviving the removal of acoustic evidence (via silence or corruption), effectively acting as a hallucination penalty; (2) an Attention Density Grounding reward that encourages the model to attend to the specific audio tokens corresponding to the time spans it claims in its output; and (3) a token-wise fine-grained advantage mechanism that assigns credit at the segment level rather than the sequence level, which is crucial for long, dense captions. This approach avoids modifying the base model's architecture or vocabulary, making it a plug-and-play improvement for existing Large Audio Language Models (LALMs). The methodology is sound, addressing the specific failure modes of LALMs on long audio (hallucination and temporal drift) with targeted RL interventions.
The authors introduce EgoSCAPE, a new benchmark derived from Ego4D with 700 manually verified audio clips up to 5 minutes long, featuring dense captions, sound sources, and temporal grounding queries. This is a significant contribution as existing benchmarks often lack dense annotations for long-form audio. Experiments show that ATTEST outperforms strong baselines (including Gemini 3.1 Pro, GPT-4o Audio, and specialized models like TimeAudio and AudioFlamingo 3) on both the new EgoSCAPE benchmark and existing datasets (Clotho-Moment, CASTELLA). The ablation studies convincingly demonstrate the individual contribution of each reward component, particularly the CAG reward in reducing hallucinations. The results on general audio understanding benchmarks (MMAR, LongAudioBench) confirm that the method does not cause catastrophic forgetting, which is a common issue with temporal grounding fine-tuning.
The paper provides a detailed description of the benchmark creation pipeline, including the use of BEATs for initial detection and LLMs for verification, followed by human verification. The RL training details (LoRA, GRPO parameters, reward weights) are specified. However, the code and the EgoSCAPE benchmark itself are not explicitly linked in the provided text (though a reproducibility statement is made). The reliance on specific proprietary models (Gemini, GPT) for baselines limits full reproducibility of those comparisons, but the open-source base models (Qwen, AudioFlamingo) allow for partial reproduction.
The benchmark is limited to 5-minute clips, which is still shorter than some real-world long-form audio scenarios (e.g., hours-long podcasts). The method relies on the availability of attention weights, which may not be accessible in all proprietary or optimized inference engines. The counterfactual reward requires generating a reference model output on corrupted audio, adding computational overhead during training. The paper acknowledges that the method is currently evaluated on audio-only tasks and does not extend to video or omni-modal temporal grounding in the main experiments.
This work addresses a critical bottleneck in deploying LALMs for real-world applications where audio is long and unstructured. By providing a rigorous benchmark (EgoSCAPE) and a training framework (ATTEST) that reduces hallucinations without architectural changes, it enables more reliable temporal reasoning in audio AI. This has implications for accessibility tools, video indexing, and autonomous agents that need to understand and locate specific events in long audio streams. The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
Existing audio deepfake detection (ADD) methods frequently exhibit limited generalization, as fixed bottleneck architectures fail to adapt to the heterogeneous information densities inherent in diverse spoofing artifacts. Moreover, standard augmentation often leads to a pathological robustness-accuracy trade-off. In this paper, we propose a unified ADD framework to address these challenges. Our approach integrates: (1) an Adaptive Audio Feature Integration (AAFI) module that dynamically selects optimal latent dimensions via a compression pool and a stochastic exploration mechanism, ensuring flexible representations across multiple types of dynamic attacks; and (2) a Mel-Band Adversarial Perturbation (Mel-BAP) strategy that applies targeted regularization in the mel-spectrogram domain to encourage the model to learn invariant decision boundaries. Extensive evaluations on benchmarks including ADD 2022, FakeOrReal, and In-the-Wild demonstrate that our model based on Whisper-small achieves a state-of-the-art average Equal Error Rate (EER) of 1.51%. Compared to existing SOTA models such as the 3-billion-parameter Resemble-Detect-3B-Omni, our framework reduces the parameter count by up to 91.6% while achieving a 60.7% improvement in average EER. In a separate cross-lingual evaluation, the Whisper-base configuration achieves EERs of 3.91-5.65% on five unseen languages. The framework also maintains strong resilience to real-world acoustic distortions. This work provides a scalable and efficient "plug-in" solution for enhancing trust and security in the audio Web ecosystem.
Primary: Imperial College London
All Institutions: Imperial College London, Taiyuan University of Technology, University of Southampton, Yangtze Delta Region Institute of Tsinghua University
The paper presents a novel adaptive compression and targeted perturbation framework for audio deepfake detection, achieving state-of-the-art performance with significantly reduced model size. The integration of dynamic feature selection and frequency-domain adversarial training offers a robust solution to the generalization challenges inherent in current ADD methods, providing a scalable and efficient tool for enhancing trust in digital audio content.
The paper proposes a unified framework for Audio Deepfake Detection (ADD) built upon the Whisper-small backbone. The core contribution is the Adaptive Audio Feature Integration (AAFI) module, which employs a heterogeneous compression pool and a stochastic router to dynamically select latent dimensions based on input characteristics. This addresses the limitation of fixed bottleneck architectures by allowing the model to adapt its information density to different types of spoofing artifacts. The second component, Mel-Band Adversarial Perturbation (Mel-BAP), applies targeted adversarial training in the mel-spectrogram domain, specifically focusing on frequency bands where neural vocoder artifacts are prevalent. The methodology is sound, leveraging the strong pre-trained representations of Whisper while introducing adaptive mechanisms to enhance generalization. The use of the Whisper decoder for classification rather than a simple MLP head is a creative choice, justified by the need for temporal granularity and semantic-acoustic alignment.
The experiments are extensive, covering ADD 2022, FakeOrReal, and In-the-Wild benchmarks. The reported state-of-the-art average EER of 1.51% is impressive, especially given the significant reduction in parameter count compared to larger models like Resemble-Detect-3B-Omni. The cross-lingual evaluation on five unseen languages demonstrates strong zero-shot transferability. The ablation studies (referenced in the text) appear to validate the individual contributions of AAFI and Mel-BAP. The comparison against SOTA models is rigorous, highlighting both accuracy and efficiency gains.
The paper provides detailed architectural descriptions, including the formulation of the routing mechanism and the loss functions. However, specific hyperparameters (e.g., noise scale, top-k value, exact compression dimensions) are deferred to the appendix. The reliance on the Whisper backbone ensures that the base model is accessible, but the specific implementation details of the AAFI module and Mel-BAP strategy would require careful replication. The code availability is not explicitly mentioned in the provided text, which is a minor drawback for immediate reproducibility.
The framework relies heavily on the Whisper encoder, which, while powerful, may still struggle with extremely low-quality or heavily distorted audio where the pre-trained features are less reliable. The stochastic exploration mechanism in AAFI adds computational overhead during training, although inference remains efficient. The paper focuses on binary classification (genuine vs. spoofed), and extending this to multi-class or open-set detection scenarios is not explored. Additionally, the performance on real-world, non-benchmarked audio sources is only partially addressed through the In-the-Wild dataset.
This work has significant implications for the security of the audio web ecosystem. By providing a scalable and efficient solution for detecting audio deepfakes, it can be integrated into content moderation systems, verification platforms, and forensic tools. The focus on generalization and robustness makes it particularly relevant for real-world deployment where spoofing techniques evolve rapidly. The efficiency gains also make it suitable for edge devices or large-scale processing, broadening its applicability. The paper presents a novel adaptive compression and targeted perturbation framework for audio deepfake detection, achieving state-of-the-art performance with significantly reduced model size. The integration of dynamic feature selection and frequency-domain adversarial training offers a robust solution to the generalization challenges inherent in current ADD methods, providing a scalable and efficient tool for enhancing trust in digital audio content.
Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing \textbf{UA-Bench}, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0\%--13.1\% leaf accuracy, against a random reference of 7.7\%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.
Primary: Southeast University
All Institutions: Southeast University
The paper introduces UA-Bench, the first large-scale benchmark for evaluating Large Audio-Language Models on underwater acoustic signal recognition, combining parameterized source synthesis with modeled propagation to enable traceable error analysis. By revealing that current LALMs perform near-randomly on fine-grained underwater classification despite some capability in coarse routing, the work provides a rigorous diagnostic framework and a necessary reference point for advancing audio understanding in specialized, non-speech domains.
The methodology is robust and well-designed for a synthetic benchmark. The authors successfully decouple source generation from channel propagation using BELLHOP, allowing for traceable error attribution. The "gated coarse-to-fine" evaluation protocol is a significant methodological contribution, as it distinguishes between failures in source routing (Active vs. Passive) and fine-grained classification, which standard accuracy metrics obscure. The inclusion of a paired silent-audio control is a rigorous diagnostic tool to verify that models are actually utilizing the waveform rather than relying on question priors or class imbalance. The synthesis pipeline, covering detection pulses, communication waveforms, and ship-radiated noise with specific physical parameters (sound-speed profiles, multipath), demonstrates a deep understanding of underwater acoustics.
The experiments are comprehensive, evaluating nine state-of-the-art open-source Large Audio-Language Models (LALMs) without task-specific adaptation. The results are sobering and informative: leaf accuracy ranges from 2.0% to 13.1%, barely above the random reference of 7.7%. The analysis correctly identifies that while some models show significant gains in coarse routing (L1) when audio is present, the fine-grained leaf-level gains are minimal (max +4.7 points). The breakdown of errors into invalid-format, cascade, and classification errors provides valuable insight into the specific failure modes of current LALMs on non-speech audio.
High. The paper provides extensive details on the synthesis parameters, partitioning strategy (source-grouped to prevent leakage), and evaluation protocol. The release of the training partition and the fixed evaluation subset (UA-Bench-Eval) ensures that future work can be compared fairly. The code for generation and evaluation is implied to be available given the nature of the benchmark release, though specific URLs are not in the text provided.
The primary limitation is the synthetic nature of the data. While the propagation is modeled, it may not capture the full complexity of real-world underwater noise (e.g., biological noise, surface clutter, non-linear effects). The paper acknowledges this and positions the benchmark as a controlled reference point rather than a field-ready solution. Additionally, the evaluation is limited to zero-shot performance; the impact of fine-tuning on these models using the provided training partition is not explored in this paper.
This paper provides a critical missing piece in the evaluation of LALMs. By demonstrating that current models struggle significantly with engineered underwater signals, it highlights the gap between general audio pretraining and specialized domain recognition. It sets a baseline for future research in domain adaptation for audio-language models and provides a tool for the maritime and ocean sensing communities to evaluate new models. The paper introduces UA-Bench, the first large-scale benchmark for evaluating Large Audio-Language Models on underwater acoustic signal recognition, combining parameterized source synthesis with modeled propagation to enable traceable error analysis. By revealing that current LALMs perform near-randomly on fine-grained underwater classification despite some capability in coarse routing, the work provides a rigorous diagnostic framework and a necessary reference point for advancing audio understanding in specialized, non-speech domains.
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: https://any-edit.github.io/. AnyEditBench: https://huggingface.co/datasets/Ririiiii/anyeditbench.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Institute of Media Technology and Experience Design (Huawei), City University of Macau, Amphion Technology Co., Ltd.
AnyEdit introduces a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments by decoupling semantic editing from acoustic rendering. The paper makes significant technical contributions, including a noise-robust prosody tokenizer, an instruction-guided autoregressive editor with unmodified-area-aware supervision, and a flow-matching acoustic model for in-context rendering, along with the release of AnyEditBench, a comprehensive benchmark for real-world vocal editing. The methodology is well-motivated, the experiments are thorough, and the results demonstrate state-of-the-art performance, making this a strong and impactful contribution to the field of audio processing.
The paper proposes a unified, separation-free framework for speech and singing voice editing that decouples semantic editing from acoustic rendering. The core innovation lies in the two-stage architecture: an autoregressive (AR) editor that operates in a clean latent space to predict prosody and content-style tokens, followed by a flow-matching transformer (FMT) that renders the final mel-spectrogram conditioned on the source acoustic context. This design elegantly avoids the artifacts associated with source separation while preserving environmental consistency. The introduction of a noise-robust prosody tokenizer via teacher-student distillation is a strong technical contribution, enabling stable prosodic conditioning even from degraded inputs. Additionally, the use of Direct Preference Optimization (DPO) on mined real-to-real singing pairs (leveraging song structure repetitions) is a creative and effective strategy for improving singing quality without relying on synthetic targets. The unmodified-area-aware supervision mechanism is a practical and effective solution to the common problem of prosody drift in localized editing.
The experimental evaluation is comprehensive and rigorous. The authors introduce AnyEditBench, a valuable new benchmark covering speech and singing in clean, simulated, and in-the-wild conditions, which fills a gap in the literature. The evaluation includes both objective metrics (WER/CER, speaker similarity, F0 correlation, background similarity, RT60/SNR error) and subjective evaluations (MOS, Fooling Rate). The results demonstrate consistent superiority over strong baselines, including Vevo2, Step-Audio-EditX, and SSR-Speech, particularly in noisy and reverberant environments. Ablation studies convincingly validate the necessity of each component, such as the noise-robust tokenizer, the staged training, and the DPO alignment. The comparison with SSR-Speech is handled well, with a clear explanation of why SSR-Speech's higher scores on some metrics are not directly comparable due to its full-utterance regeneration nature.
The paper provides sufficient details for reproduction, including model architectures, training strategies, data construction pipelines, and evaluation metrics. The release of AnyEditBench on Hugging Face is a significant contribution to reproducibility and community benchmarking. However, the specific implementation details of the AR editor and FMT are somewhat high-level, and the exact hyperparameters for the DPO training and the teacher-student distillation are not fully detailed in the main text (though likely in the appendix). The use of standard components like Qwen2.5 and LLaMA-2 blocks aids in reproducibility.
The primary limitation is the reliance on a two-stage cascaded architecture, which may introduce latency and potential error propagation from the AR stage to the FMT stage. The model's performance on extremely low SNR conditions (e.g., below 0 dB) is not extensively explored. Additionally, the framework is currently limited to Chinese and English, and its generalization to other languages or more complex polyphonic backgrounds remains to be seen. The DPO mining strategy, while clever, is specific to popular music structures and may not generalize to all types of singing or non-musical vocal performances.
This work has significant potential impact on audio post-production, content creation, and accessibility tools. By enabling high-quality, localized editing of real-world recordings without the need for source separation, it simplifies the workflow for creators and reduces computational overhead. The unified handling of speech and singing opens up new possibilities for music production and interactive media. The release of a comprehensive benchmark will facilitate future research in this area and enable fair comparison of different approaches. AnyEdit introduces a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments by decoupling semantic editing from acoustic rendering. The paper makes significant technical contributions, including a noise-robust prosody tokenizer, an instruction-guided autoregressive editor with unmodified-area-aware supervision, and a flow-matching acoustic model for in-context rendering, along with the release of AnyEditBench, a comprehensive benchmark for real-world vocal editing. The methodology is well-motivated, the experiments are thorough, and the results demonstrate state-of-the-art performance, making this a strong and impactful contribution to the field of audio processing.
Text-conditioned symbolic music generation has attracted growing interest with the development of generative models. Yet two questions remain underexplored: what aspects of music the conditioning text should describe, and how the corresponding data should be collected and trained to improve controllability and text-music alignment. We address these challenges with TASTE, a taxonomy-aware framework for symbolic music generation with a main focus on piano pieces. First, we introduce a taxonomy of musical elements that specifies how a piece can be captured through textual description in a musically grounded way. The taxonomy comprises a set of metadata tags (genre, emotion, section plan, etc.) together with a free-form description of music imagery. This structure decomposes textual input to match the generation goal more coherently and precisely. Second, we build a data pipeline for collecting, annotating, and retrieving public symbolic piano resources, yielding a corpus of 432K MIDI files labeled at multiple levels of granularity. Finally, we design a staged training procedure that leverages this multi-granularity data across two model families: an autoregressive model and a non-autoregressive diffusion model. Both objective and subjective evaluations show that our models achieve superior musical-element control, text-description alignment, and overall music quality. We will release the full pipeline, all annotated datasets, and the model checkpoints.
Primary: University of California, San Diego
All Institutions: University of California, San Diego, University of Rochester
The paper introduces a taxonomy-aware framework for symbolic piano generation that effectively bridges the gap between user intent and model control. By decomposing musical descriptions into structured tags, section plans, and imagery, and building a robust multi-tier data pipeline to annotate 432K MIDI files, the authors demonstrate significant improvements in attribute agreement, temporal structure, and text-music alignment over existing baselines. The rigorous evaluation, including both objective metrics and human listening studies, validates the effectiveness of the proposed staged training procedure for both autoregressive and diffusion models, establishing a new standard for controllable symbolic music generation.
The paper proposes TASTE, a framework that addresses the "description-intention gap" in text-conditioned symbolic music generation. The core methodological contribution is a taxonomy that decomposes musical description into three distinct channels: (1) objective tags (key, tempo, emotion, genre, etc.), (2) section plans (temporal structure), and (3) free-form imagery descriptions. This decomposition allows for targeted annotation and evaluation. The data pipeline is sophisticated, utilizing a multi-tier approach (high-quality human-annotated, curated web-search/LLM-annotated, and large-scale weakly-supervised) to build a corpus of 432K MIDI files. The model architecture integrates these channels into both an Autoregressive (AR) transformer and a Diffusion Transformer (DiT). The AR model uses AdaLN-Zero for tag conditioning and cross-attention for imagery, while the DiT uses similar conditioning mechanisms in a latent space. The staged training procedure, which pretrains on large-scale data with estimated attributes before fine-tuning on high-quality labeled data, is a sound strategy for leveraging weak supervision.
The evaluation is comprehensive, covering objective metrics (tag agreement, section boundary F1, FAD, FMD, Audiobox scores) and subjective listening studies. The results show that TASTE models outperform baselines like MuseCoco, text2midi, and MIDI-LLM in both attribute control and text-music alignment. Notably, the DiT model achieves superior temporal structure realization (boundary F1 of 0.868) compared to the AR model (0.517), suggesting that the non-autoregressive approach handles long-range structural dependencies better. The listening study confirms that listeners can match TASTE outputs to their intended imagery descriptions significantly better than chance and even better than reference performances in some cases, indicating strong semantic alignment.
The paper provides extensive details on the data pipeline, annotation protocols, model architectures, and training configurations. The authors commit to releasing the full pipeline, annotated datasets, and model checkpoints. The use of standard components (Qwen3, CLaMP3, Aria embeddings) and clear descriptions of the weak supervision strategies (probes, retrieval, rewriting) enhance reproducibility. The detailed appendices on probe calibration and confidence gates are particularly helpful for replicating the data construction process.
The primary limitation identified is the reliability of labels for "accompaniment" and "performance style," which remain the hardest fields to control due to lower precision in the weak supervision pipeline. The evaluation is focused on piano music, limiting generalizability to other instruments or full orchestral scores. The reliance on LLMs for annotation (web search and audio description) introduces potential biases and inconsistencies, although mitigated by the multi-tier verification process. The AR model's weaker performance in temporal structure realization compared to DiT is a notable weakness for that specific architecture.
This work contributes to the field of controllable music generation by providing a principled framework for describing and controlling musical attributes. The taxonomy and data pipeline can be extended to other musical domains. The release of a large, multi-granularity annotated dataset will benefit the research community by enabling fairer comparisons and more fine-grained control in future models. The integration of symbolic and perceptual conditioning offers a path toward more human-like interaction with music generation systems. The paper introduces a taxonomy-aware framework for symbolic piano generation that effectively bridges the gap between user intent and model control. By decomposing musical descriptions into structured tags, section plans, and imagery, and building a robust multi-tier data pipeline to annotate 432K MIDI files, the authors demonstrate significant improvements in attribute agreement, temporal structure, and text-music alignment over existing baselines. The rigorous evaluation, including both objective metrics and human listening studies, validates the effectiveness of the proposed staged training procedure for both autoregressive and diffusion models, establishing a new standard for controllable symbolic music generation.
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.
Primary: Cornell University
All Institutions: Cornell University
The paper introduces a novel synthetic data generation pipeline for ultrasonic silent speech recognition, significantly improving WER through a two-stage GAN and cross-modal contrastive learning. The rigorous ablations and controls demonstrate the specific value of the synthetic data, establishing a new state-of-the-art for minimally obtrusive silent speech interfaces.
The paper proposes SoniGAN, a two-stage generative framework for synthesizing ultrasonic echo profiles from text. The methodology is technically sound and well-motivated. It leverages a voice-cloned TTS model (F5-TTS) to generate audio from text, converts this to soft speech units to strip timbre, and then uses a conditional GAN architecture to map these units to ultrasonic signals. The two-stage approach (envelope first, then fringe structure) is a clever solution to the high-frequency nature of ultrasonic fringes, which are difficult for single-step generators to capture. The use of a frozen recognizer as a perceptual loss is a strong choice to ensure semantic fidelity. The inclusion of cross-modal contrastive learning (crossCon) in the recognizer training is a significant methodological contribution that effectively utilizes the paired voiced/silent data available in the benchmark.
The experiments are rigorous and comprehensive. The authors provide a clear ablation study showing the contribution of each component (synthetic pretraining, faulty session handling, decoding strategies). The controls for "extra optimizer steps" vs. "data content" are particularly well-designed, proving that the gain comes from the synthetic data's content rather than just increased training time. The results show a significant improvement in Word Error Rate (WER), moving from a baseline of ~26% to 14.9%, which is a substantial leap for this modality. The evaluation on the public SoniSpeech benchmark ensures comparability.
The paper provides detailed hyperparameters, architecture descriptions, and training procedures. However, the specific implementation of the "soft speech units" extraction and the exact configuration of the F5-TTS cloning process are not fully detailed, which might pose challenges for exact reproduction. The code is not explicitly linked in the provided text, though the use of standard components (ResNet, Transformer, F5-TTS) aids in reconstruction.
The system relies on a specific benchmark (SoniSpeech) and a specific TTS model. The generalizability to other ultrasonic sensing setups or different languages is not tested. The "faulty microphone" augmentation is specific to the hardware failure mode of the benchmark device. The 14.9% WER, while a major improvement, is still higher than typical audible speech recognition systems, indicating room for further improvement.
This work has significant implications for accessibility and human-computer interaction. Silent speech interfaces are crucial for individuals with vocal impairments or for use in noisy environments. By reducing the data collection burden via synthetic supervision, this approach makes it easier to scale silent speech systems to open vocabularies, potentially enabling more natural and less obtrusive communication interfaces. The paper introduces a novel synthetic data generation pipeline for ultrasonic silent speech recognition, significantly improving WER through a two-stage GAN and cross-modal contrastive learning. The rigorous ablations and controls demonstrate the specific value of the synthetic data, establishing a new state-of-the-art for minimally obtrusive silent speech interfaces.
Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.
Primary: ETH Zurich
All Institutions: ETH Zurich, EPFL, CSCS
The paper presents a comprehensive and well-executed approach to extending a text LLM to a multimodal one using discrete tokenization, resulting in a fully open model family that is competitive with other open models. The detailed methodology, extensive experiments, and high reproducibility make it a significant contribution to the field of multimodal LLMs.
The paper proposes a robust methodology for extending a text-only LLM to a multimodal one using discrete tokenization rather than continuous encoders. By utilizing the Emu3.5 image tokenizer and WavTokenizer for audio, the authors integrate new modalities into the existing vocabulary, allowing the standard next-token prediction objective to apply to all data types. This approach simplifies the training infrastructure, as it avoids the computational overhead of running separate vision/audio encoders during the forward pass. The training recipe is carefully staged: Stage 1 focuses on unpaired multimodal data to learn representation, while Stage 2 introduces paired data to align modalities with text, restricting the loss to text tokens to prevent the model from generating non-text outputs. This design choice is well-motivated and supported by ablation studies showing that text-only targets in the paired stage yield stronger image understanding. The extension of the context window to 262k tokens is also handled systematically through a multi-step extension process with adjusted rotary position embeddings.
The experimental evaluation is extensive, covering text, image, and audio benchmarks. The models are compared against other open and open-weight models, demonstrating competitive performance, particularly in document understanding and lower-resource speech recognition. The paper includes ablations on the choice of image tokenizer (Emu3 vs. Emu3.5) and the two-stage training recipe, providing insight into the trade-offs between reconstruction quality and token efficiency. The evaluation of the thinking mode shows a significant lift in mathematical reasoning, validating the effectiveness of the post-training pipeline. However, the paper acknowledges that specialized vision and audio models still outperform Apertus 1.5 on high-resource perception and transcription tasks, which is a fair and honest assessment of the current limitations.
The paper excels in reproducibility, adhering to the "fully open" philosophy. All model weights, intermediate checkpoints, data preparation pipelines, training code, and evaluation harnesses are released under permissive licenses. The detailed documentation of the data mixture, including source versions and licenses, allows for full auditability. The release of the tokenizer benchmarks and the specific hyperparameters for each training stage further enhances the ability for other researchers to reproduce the results. The use of standard frameworks like Megatron-LM, with clearly documented modifications, lowers the barrier to entry for replication.
The primary limitation is the performance gap with specialized vision and audio models on high-resource tasks. The discrete tokenization approach, while efficient, results in a coarser representation of images, which negatively impacts OCR and fine-grained detail tasks. The audio codec, while efficient, may not capture all nuances of high-quality audio, potentially limiting performance in specialized audio tasks. The paper also notes that the 70B model was trained on fewer tokens than intended due to resource constraints, which may affect its final performance relative to the 8B model.
The release of Apertus 1.5 has significant broader impact by providing a fully open alternative to proprietary multimodal models. This supports research and development in AI by enabling independent study, deployment, and modification of the models. The emphasis on open data and values aligns with ethical AI development practices, promoting transparency and accountability. The models' multilingual capabilities and cultural benchmarks contribute to more inclusive AI systems. The open release of the training infrastructure and data pipelines can serve as a reference for other institutions aiming to develop open multimodal models. The paper presents a comprehensive and well-executed approach to extending a text LLM to a multimodal one using discrete tokenization, resulting in a fully open model family that is competitive with other open models. The detailed methodology, extensive experiments, and high reproducibility make it a significant contribution to the field of multimodal LLMs.
Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.
Primary: MIT (Inferred from "MIT-IBM Watson AI Lab" and common affiliation patterns for this research group; specific author list missing in text but "MIT-IBM" is explicit)
All Institutions: MIT, IBM Research, Hyundai Motor Company (Supporters)
The paper demonstrates that harmonic decisions in a symbolic music transformer are causally localized in middle layers and can be manipulated via latent steering to control cadence resolution. By combining causal interventions, linear probing, and representation steering on Bach chorales, the authors reveal that the model's internal geometry mirrors music theory (with relative keys closer than parallel keys) and that these representations are actionable for inference-time control, offering a new pathway for interpretable and controllable symbolic music generation.
The paper employs a rigorous mechanistic interpretability framework applied to a symbolic music transformer (Aria). The methodology is sound, utilizing three complementary techniques: (1) Causal interventions (activation patching/interchange interventions) to localize harmonic computation, (2) Linear probing to assess decodability of harmonic attributes (mode, key), and (3) Representation steering to test actionability. The construction of counterfactual pairs via minimal single-note edits to Bach chorales is a clever and effective way to isolate specific harmonic decisions (cadence resolution) from other musical features. The derivation of steering vectors from probe weights is a standard but effective technique in this domain. The analysis of the geometric relationship between relative and parallel keys in the latent space provides a strong theoretical link between model internals and music theory.
The experiments are well-designed and comprehensive. The use of Bach chorales is appropriate for this specific query (harmonic structure), though it limits generalizability. The evaluation metrics are robust, combining structural integrity checks (error rates, repetition), perceptual quality (FMD), and harmonic specificity (Roman numeral entropy, top-1 dominance). The results clearly demonstrate that harmonic representations are localized in middle layers and that steering these representations effectively alters the harmonic outcome (e.g., shifting from authentic to deceptive cadences) without degrading overall musical coherence. The finding that parallel key steering is more effective than relative key steering due to latent space geometry is a significant and insightful result.
The paper provides sufficient detail for reproducibility. It specifies the model (Aria), the dataset (Bach chorales), the tool for harmonic analysis (music21), and the specific intervention methods (layers, alpha values, probe training details). The code and data are available on the project webpage. The specific layer ranges for intervention (5-10 for mode, 11-15 for key) are clearly stated.
The primary limitation is the narrow scope of the dataset (Bach chorales). The authors acknowledge that whether these findings hold for more chromatic or less conventional music is an open question. The model used (Aria) is a specific LLaMA-based transformer; results may not generalize to other architectures (e.g., diffusion models for audio, or different tokenization schemes). The steering is limited to harmonic resolution at cadences; it is unclear if this method can control other musical aspects like melody or rhythm with similar precision.
This work contributes to the growing field of mechanistic interpretability in generative models, specifically for music. It demonstrates that high-level musical concepts are not just distributed noise but are structured and manipulable. This has implications for controllable music generation, allowing users to steer harmonic progressions without retraining. It also provides insights into how neural networks internalize symbolic rules, bridging the gap between music theory and machine learning. The paper demonstrates that harmonic decisions in a symbolic music transformer are causally localized in middle layers and can be manipulated via latent steering to control cadence resolution. By combining causal interventions, linear probing, and representation steering on Bach chorales, the authors reveal that the model's internal geometry mirrors music theory (with relative keys closer than parallel keys) and that these representations are actionable for inference-time control, offering a new pathway for interpretable and controllable symbolic music generation.
Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.
Primary: AutoArk
All Institutions: AutoArk
The paper presents Audio8-TTS, a compact 0.6B parameter zero-shot voice cloning model that achieves state-of-the-art perceptual naturalness by employing a reference-based realism critic trained on a quality ladder of open-source TTS systems, integrated into a Group Relative Policy Optimization (GRPO) framework to optimize prosodic realism alongside intelligibility and speaker similarity.
The paper proposes a three-stage training framework for a compact (0.6B) dual-autoregressive TTS model. The core novelty lies in the third stage: using a learned "realism critic" as a reward model for Group Relative Policy Optimization (GRPO). The critic is trained on a "quality ladder" constructed from outputs of various open-source TTS systems, ranked via a private blind listening study, with human recordings as the top anchor. A key technical contribution is the "identity-erasure" objective, which uses pause-compression degradation to prevent the critic from relying on generator-specific fingerprints rather than prosodic quality. The architecture itself (fast-slow dual autoregressive) is standard in recent codec-LM literature, but the specific application of reference-based realism rewards to compact models is a distinct methodological contribution.
The evaluation is comprehensive, covering 9 CV3 languages and 13 additional languages. The model achieves competitive WER and SIM scores compared to much larger models (e.g., Seed-TTS, MiniMax). The most significant evidence is the blind human evaluation, where the realism-optimized model received a substantially higher Elo rating than the SFT and WER/SIM-only models, validating the effectiveness of the realism critic. The ablation of the critic's generalization to unseen systems (Fish Audio S2, Spark-TTS) is a strong point, showing the critic learns quality rather than memorizing specific generators.
The authors release code and model weights on GitHub. They provide detailed descriptions of the data mixture, training stages, and critic construction. However, the "private blind listening study" and "private collection" of data limit full reproducibility of the critic's training data. The use of specific open-source systems for the quality ladder is reproducible, but the exact ranking and degradation parameters are proprietary.
The realism critic requires a matched human reference for every sentence during inference/evaluation, which is a significant practical constraint for zero-shot cloning where such references may not exist. The paper acknowledges that the critic's signal is primarily driven by pause structure due to the degradation method used, potentially neglecting other prosodic cues like pitch and energy. Additionally, the model struggles with low-resource languages (e.g., Thai, Finnish) where error rates remain high.
This work demonstrates that compact TTS models can achieve high perceptual naturalness through targeted preference optimization, not just scale. The approach of using a learned critic to bridge the gap between objective metrics (WER/SIM) and subjective realism is applicable to other generative audio tasks. The identity-erasure technique is a valuable contribution to the field of reward model design for generative models. The paper presents Audio8-TTS, a compact 0.6B parameter zero-shot voice cloning model that achieves state-of-the-art perceptual naturalness by employing a reference-based realism critic trained on a quality ladder of open-source TTS systems, integrated into a Group Relative Policy Optimization (GRPO) framework to optimize prosodic realism alongside intelligibility and speaker similarity.
Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.
Primary: Brown University
All Institutions: Brown University, MIT, Yale University, The University of Hong Kong
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Sonic-AMG introduces a novel framework for streaming human motion generation from spatial audio, addressing key limitations of prior offline methods. The core methodology is built upon a causal transformer with a flow head, operating on continuous motion latents from a pretrained VAE. A significant methodological contribution is the separation of acoustic content ("what") and body-relative source geometry ("where") into distinct conditioning tokens. This design choice, supported by ablations, allows the model to better disentangle and utilize these modalities. The 11-dimensional spatial descriptor, inspired by perceptual research, is a thoughtful design for capturing relevant spatial cues like proximity, relative motion, and looming. The paper effectively tackles two critical training-inference mismatches inherent in autoregressive streaming generation. First, the "history mismatch" (teacher-forced training vs. generated histories at inference) is addressed by "rehearsal training." This innovative approach combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. This hybrid strategy allows the model to learn stable long-range dependencies while also gaining robustness to accumulated errors from its own predictions. Second, the "spatial conditioning mismatch" (ground-truth SSL vs. generated root poses) is resolved by "spatial feedback." This mechanism recomputes body-relative source locations from generated root poses during both rehearsal training and inference, ensuring consistency between the generated motion and the spatial conditions. The integration of these components within a causal transformer and flow-matching framework is technically sound and well-justified. The use of classifier-free guidance during inference further enhances generation quality.
The experimental evaluation is comprehensive and well-structured, primarily using the SAM dataset. The paper establishes state-of-the-art performance for Sonic-AMG on FID and R-precision metrics, demonstrating both distributional fidelity and conditioning alignment. A crucial aspect of the evaluation is the distinction between "GT SSL" (ground-truth spatial conditioning, for fair comparison with baselines) and "Online SSL" (spatial feedback from generated motion, reflecting real-world streaming). Sonic-AMG maintains strong performance under the more challenging Online protocol, which is a significant achievement. The ablation studies are particularly strong, providing clear evidence for the efficacy of the proposed components: 1. **Rehearsal training vs. two-forward training:** Rehearsal training consistently outperforms various two-forward schedules, validating the benefit of recursive history construction. 2. **Role of teacher-forced supervision:** Demonstrates that combining self-rollouts with full-sequence supervision is crucial for fidelity and alignment. 3. **Effect of spatial feedback:** Shows that training with spatial feedback dramatically improves performance under the Online protocol, confirming its necessity for consistent spatial conditioning. 4. **Separate spatial tokens:** Ablation against a "Fused" approach confirms the advantage of distinct acoustic and spatial tokens. 5. **Learned acoustic features:** Shows that simpler, learned log-mel features outperform MOSPA's high-dimensional audio descriptors, supporting the hypothesis that separating "what" from "where" helps the acoustic branch focus on content. Beyond quantitative metrics, the paper provides compelling qualitative results, including examples of recognizable and spatially appropriate reactions, and the ability to generate coherent motion across complete audio clips and successive sound events. The demonstration of real-time streaming efficiency, with significantly reduced initial response latency compared to MOSPA, is a strong practical contribution. The inclusion of physical simulation retargeting to a Unitree G1 robot further showcases the robustness and applicability of the generated motions. The "No-SSL" variant, which derives spatial tokens directly from binaural audio, is an excellent exploration of alternative inference scenarios, demonstrating the model's flexibility and potential for deployment without explicit source location inputs.
The paper includes a dedicated "Reproducibility statement" and detailed appendices covering implementation details, training protocols, and runtime measurements. This includes specifics on architecture (transformer layers, heads, width, flow head details), optimization (AdamW, learning rate, batch size), training hardware and time, motion autoencoder pretraining, acoustic and spatial encoding, guidance, and training sequence construction for both teacher-forced and rehearsal training. The authors commit to releasing code and models upon acceptance, which, combined with the extensive details provided, suggests high reproducibility.
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Primary: FAIR, Meta Superintelligence Labs
All Institutions: The University of Texas at Austin, FAIR, Meta Superintelligence Labs
The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
The paper proposes RUBATO, a multistream positional encoding mechanism that assigns independent "clocks" to different data streams (real-time audio vs. internal thinking) in full-duplex speech models. The core innovation is decoupling the positional indices of asynchronous streams to prevent temporal distortion when the decoding rate of the thinking stream varies. The authors derive three variants: TM-RoPE (subspace partitioning), Key-Clock (conditional rotation based on key token clock), and Full-Clock (rotating all dimensions by the key's clock). The methodology is mathematically sound, leveraging the properties of Rotary Position Embeddings (RoPE) to allow zero-shot interpolation of inference speeds. The implementation strategy, which expands hidden dimensions during training to fit standard FlashAttention kernels and uses separate KV caches during inference, is a clever engineering solution that avoids custom Triton kernels, enhancing practicality.
The experiments are rigorous, evaluating the model on Spoken GSM8K and BIG-Bench Audio, including a novel "Overlapped" scenario where the model must handle interruptions while reasoning. The results demonstrate that RUBATO-Full maintains high accuracy (48-51%) across varying decoding budgets, whereas baselines collapse to near-zero accuracy at interpolated speeds. The paper provides strong evidence for the utility of the method in reducing time-to-answer by leveraging faster hardware without retraining. The inclusion of UTMOS and Speaker Similarity metrics for filler audio quality adds depth to the evaluation, confirming that the architectural changes do not degrade acoustic quality.
The paper provides a detailed reproducibility statement, including closed-form equations for the attention mechanisms and clear descriptions of the training data synthesis process. The use of standard backbones (Gemma-4) and tokenizers (NanoCodec) aids reproducibility. However, the specific hyperparameters for the training mixture and the exact construction of the synthetic datasets are deferred to the appendix, which is not fully visible in the provided text, though the main text describes the logic clearly.
The primary limitation is the reliance on synthetic data for training, as no suitable full-duplex spoken reasoning corpora exist. The evaluation is limited to mathematical and logical reasoning tasks (GSM8K, BIG-Bench), which may not generalize to all conversational domains. Additionally, the method assumes a specific interleaving strategy (SCoT), and its applicability to other full-duplex architectures (e.g., concatenation-based) is not explored. The "Full-Clock" variant, while best performing, requires duplicating the head dimension, which increases memory usage.
This work has significant implications for the deployment of real-time AI agents. By decoupling reasoning speed from real-time constraints, it enables more efficient use of heterogeneous hardware. The concept of independent clocks for asynchronous streams is a generalizable architectural pattern that could apply to other multimodal or multi-agent systems requiring variable-rate processing. It addresses a critical bottleneck in making LLMs truly interactive and responsive in speech-based interfaces. The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.
Primary: Eindhoven University of Technology
All Institutions: Eindhoven University of Technology
AudioTETRIS introduces a training-free token compression framework for LALMs that leverages encoder-state dynamics as a phoneme-transition prior to guide attention-based pruning. The paper demonstrates that preserving acoustically rich transition regions is crucial for maintaining high accuracy under aggressive compression, outperforming existing baselines across multiple backbones and tasks while significantly reducing inference compute.
The paper proposes AudioTETRIS, a training-free token compression framework for Large Audio Language Models (LALMs). The core innovation is the derivation of a "phoneme-transition-aware" acoustic prior from the temporal derivative of the audio encoder's hidden states. The authors hypothesize that regions of rapid spectral transition in the encoder space correspond to phonetic transitions, which carry high information density. This prior is fused with a depth-adaptive attention score from the LLM decoder to guide progressive token pruning. The methodology is sound, leveraging existing model components (encoder hidden states, decoder attention) without requiring fine-tuning. The validation of the acoustic prior via correlation with forced alignments and kurtosis analysis is a strong methodological addition that grounds the heuristic in linguistic reality.
The experiments are comprehensive, evaluating three distinct backbones (Qwen2-Audio-7B, Phi-4-Multimodal, Audio Flamingo 3) across four diverse tasks (ASR, Emotion Recognition, Audio QA, Translation). The results demonstrate that AudioTETRIS retains near-lossless performance at moderate compression rates and significantly outperforms baselines (SpeechPrune, SparseVLM, LTBM) at aggressive compression rates (75% token removal). The ablation studies effectively isolate the contributions of the acoustic prior and the attention score, confirming their complementary nature. The efficiency analysis provides concrete FLOP and latency savings, validating the practical utility of the method.
The paper provides high reproducibility standards. It includes detailed algorithmic summaries, specifies the exact layers for pruning and prior computation, and describes the calibration procedure for the crossover layer. The authors commit to releasing the full implementation and evaluation scripts. The use of standard, publicly available datasets and checkpoints further enhances reproducibility.
The method requires a one-time calibration pass per backbone to determine the crossover layer, which limits its "zero-shot" applicability to unseen architectures. The acoustic prior assumes specific geometric properties of encoder hidden states that may not hold for all encoder types (e.g., HTSAT-style event encoders). The evaluation is limited to English-source benchmarks, and performance on noisy or multilingual audio remains untested. The progressive pruning within the decoder incurs a slight TFLOP overhead compared to single-shot prefill pruning, though this is offset by accuracy gains.
This work addresses a critical bottleneck in deploying LALMs: the quadratic scaling of self-attention with long audio token sequences. By providing a training-free, plug-and-play solution that preserves fine-grained acoustic details, AudioTETRIS enables more efficient and cost-effective deployment of speech-text models. The insight that encoder representation geometry can serve as a structural prior for compression is broadly applicable to other multimodal settings where modality-specific encoders produce redundant token sequences. AudioTETRIS introduces a training-free token compression framework for LALMs that leverages encoder-state dynamics as a phoneme-transition prior to guide attention-based pruning. The paper demonstrates that preserving acoustically rich transition regions is crucial for maintaining high accuracy under aggressive compression, outperforming existing baselines across multiple backbones and tasks while significantly reducing inference compute.
Recently, generative neural synthesizers have enabled zero-shot instrument cloning from reference recordings and timbre generation from text prompts. However, existing systems generate complete audio segments offline, precluding their use as playable instruments. We present LiveSynth, a streaming neural synthesizer with flexible timbre control and interactive generation capabilities, including zero-shot instrument cloning and text-to-instrument synthesis. A feedback-free causal Transformer conditioned on MIDI events and a CLAP timbre embedding outputs continuous audio latents at 100 Hz, which a causal variational autoencoder decodes into audio. Frame-wise timbre conditioning enables continuous timbre morphing, while MIDI-conditioning dropout enables autonomous continuation from a performed prefix. On the NSynth and Slakh datasets, LiveSynth achieved timbre similarity and spectral reconstruction quality comparable to offline baselines. On an M3 MacBook Air, LiveSynth required 8.8 ms of computation per 10 ms audio frame (an RTF of 0.88) with 10 ms algorithmic latency, enabling real-time interaction with a performer.
Primary: Seoul National University
All Institutions: Seoul National University
LiveSynth introduces a streaming neural synthesizer capable of real-time zero-shot instrument cloning and text-to-instrument synthesis with 10 ms latency. By leveraging a feedback-free causal Transformer and frame-wise timbre conditioning, it achieves competitive quality with offline baselines while enabling interactive features like continuous timbre morphing and autonomous performance continuation, marking a significant step toward playable generative audio models.
The paper proposes LiveSynth, a streaming neural synthesizer that adapts the feedback-free causal Transformer framework from LiveBand for instrument cloning and text-to-instrument synthesis. The core methodological contribution is the integration of frame-wise timbre conditioning (using CLAP embeddings) with a causal VAE decoder operating at 100 Hz. The use of MIDI-conditioning dropout to enable autonomous continuation from a performed prefix is a clever training trick that facilitates interactive "call-and-response" scenarios. The decision to use continuous latents rather than quantized tokens is well-justified by the need for low latency and high reconstruction quality, avoiding the error accumulation and quality degradation associated with autoregressive token generation in strict causal settings. The Procrustes alignment for text-to-audio embedding mapping is a practical and effective solution to the modality gap in CLAP embeddings.
The experimental evaluation is rigorous, comparing LiveSynth against strong offline baselines (TokenSynth, CTD) on both NSynth and Slakh datasets. The metrics used (MERT, PANNs, MSS, F1, CE, PQ) provide a comprehensive view of timbre similarity, spectral quality, and perceptual aesthetics. The real-time inference benchmark on an M3 MacBook Air is highly relevant and convincing, demonstrating an RTF of 0.88 with 10 ms latency, which is critical for the "playable instrument" claim. The ablation on timbre morphing and autonomous continuation provides good evidence for the specific interactive capabilities. However, the lack of human listening tests (MOS/ABX) is a significant gap for a paper claiming perceptual quality improvements.
The paper provides detailed architectural specifications (parameter counts, hop sizes, attention windows) and training hyperparameters. The code and demo are available via the provided URL. The use of standard datasets (NSynth, Slakh) and open-source components (CLAP, Vocos, DAC) enhances reproducibility. However, the specific implementation of the Procrustes alignment and the exact configuration of the adversarial training might require careful attention to replicate exactly.
The primary limitation is the lack of subjective human evaluation (listening tests), which is standard for audio generation papers. The paper acknowledges lower note accuracy (F1) and production quality (PQ) compared to some offline baselines, which is a trade-off for real-time performance. The rhythmic dependence in autonomous continuation is weak, as noted by the high IOI-JSD distance. The system is currently limited to mono audio and specific instrument types found in the training data.
This work bridges the gap between offline generative audio models and real-time interactive music tools. It enables new forms of human-AI musical interaction, such as live jamming with AI-generated instruments that can morph timbres or continue performances autonomously. This has implications for music production, education, and performance art. The streaming architecture could be extended to other domains requiring real-time conditional generation. LiveSynth introduces a streaming neural synthesizer capable of real-time zero-shot instrument cloning and text-to-instrument synthesis with 10 ms latency. By leveraging a feedback-free causal Transformer and frame-wise timbre conditioning, it achieves competitive quality with offline baselines while enabling interactive features like continuous timbre morphing and autonomous performance continuation, marking a significant step toward playable generative audio models.