Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns how, when, and whether the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.
Primary: Alibaba Group
All Institutions: Alibaba Token Foundry, Alibaba Group
The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
The paper proposes a comprehensive framework for real-time voice agents, structured around three layers: Think (foundation post-training), Act (agentic tool use), and Speak/Coordinate (conversational policy). The "Think" layer introduces a sophisticated post-training pipeline combining Core-Cocktail SFT with a novel "M$^2$-OPD" (Multimodality and Multi-Teacher On-Policy Distillation) strategy. This involves using a Text Teacher and a frozen Audio Reference to supervise student-generated trajectories, effectively transferring text-based reasoning capabilities to native audio models while preserving audio-specific nuances. The "Act" layer is particularly strong, introducing self-evolving executable environments for Group Relative Policy Optimization (GRPO). By using code agents to build, validate, and evolve tasks based on model performance (difficulty gating), the system learns robust tool use, state verification, and grounded progress communication. The "Speak/Coordinate" layer formalizes the decision-making process for when to speak, act, or remain silent, addressing critical issues in full-duplex interaction like background speech handling and turn-taking.
The evaluation is extensive, covering intelligence (audio reasoning, multilingual ASR, long-context), action (tool use, retrieval), interaction (persona, empathy, full-duplex behavior), and safety. Key results include a significant improvement in task success on the $\tau$-Voice benchmark (78.4% to 82.0%) and a dramatic reduction in response rate to background speech on Full-Duplex-Bench (73.0% to 13.0%), indicating much better conversational etiquette. The model also shows strong gains in multilingual audio understanding (BBA) and safety metrics. The inclusion of a "Voice Harness" prototype for persistent tasks adds depth to the system-level contribution.
The paper provides detailed descriptions of the training pipeline, data scales (approx. 1M hours), and evaluation protocols. However, as a technical report from a major industry lab, specific hyperparameters, exact dataset compositions, and code are not fully released, limiting independent reproduction. The reliance on in-house benchmarks (Long-AMC, WebSearch1K, VoiceChat) also restricts external verification.
The primary limitation is the lack of open-source code and data, making it difficult for the community to replicate the results. The evaluation relies heavily on in-house benchmarks and automatic judges (e.g., GPT-4o-mini, Qwen-Plus), which may introduce bias. The "Voice Harness" is presented as a prototype and is not fully integrated into the core model evaluation. Additionally, the paper focuses on a specific adaptation of $\tau$-Voice (half-duplex S2T) rather than the official full-duplex S2S protocol, which may limit direct comparability with other full-duplex systems.
This work significantly advances the state-of-the-art in real-time voice assistants by integrating robust agentic capabilities with natural conversational policies. The M$^2$-OPD technique and self-evolving environment for GRPO are valuable contributions to the broader field of multimodal LLM post-training. The focus on safety and reliable interaction (e.g., handling background speech, refusing unsupported requests) is crucial for the deployment of such systems in real-world scenarios. The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Primary: Alibaba Group
All Institutions: The Chinese University of Hong Kong, Alibaba Group, Shanghai Jiao Tong University, Shanghai Innovation Institute, Zhejiang University
The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
The paper introduces a coherent framework for "native" audio-visual dialogue, addressing the critical gap in data availability for models that process raw audio/video without ASR intermediaries. The core methodological contribution is OmniVChat-Studio, a multi-agent synthesis engine. The design is robust, utilizing distinct agents (Director, Renderer, Reviewer, Validator) to handle the complex pipeline of script generation, video rendering, and quality verification. The separation of "Fixed" and "Flexible" modules allows for extensibility to new subcategories without re-engineering the core pipeline. The proposed OmniVChat-RL reward design is particularly strong, moving beyond simple correctness to incorporate efficiency (conciseness) and style (naturalness), which are crucial for conversational agents. The use of GSPO (Group Sequence Policy Optimization) with LoRA is a practical and effective training strategy for large multimodal models.
The experimental setup is rigorous, featuring a synthetic benchmark (OmniVChat-Bench) and a human-recorded transfer probe (OmniVChat-Bench-Human). The evaluation metrics are well-defined, using LLM-based grading with tiered rubrics to handle the open-ended nature of dialogue. The results show significant improvements in both correctness and efficiency (reply length reduction) when using the proposed RL method. The comparison against state-of-the-art models (Gemini, Qwen, etc.) provides a strong baseline. The ablation studies effectively isolate the contributions of the efficiency and style rewards. The transfer from synthetic to human data is a key validation point, and the results support the claim that synthetic data can effectively train models for real-world scenarios.
The paper provides detailed descriptions of the agent roles, sampling strategies, and reward functions. The code and data are promised to be released, which is standard for such contributions. The specific prompts for the Director and Reviewer agents are referenced in appendices, which aids reproducibility. However, the reliance on specific proprietary video generation models (implied by the "Renderer" agent) and LLMs for grading may limit full reproducibility for external groups without access to these specific tools.
The primary limitation is the reliance on synthetic data for training and evaluation. While the paper argues for the validity of this approach, the "reality gap" between synthesized and real-world audio-visual noise, lighting, and human behavior remains a concern, even with the human-recorded probe. The evaluation relies heavily on LLM-as-a-judge, which can introduce biases or inconsistencies, although the paper attempts to mitigate this with detailed rubrics and agreement checks. The multi-turn evaluation is limited to single-turn probes in the human dataset, potentially underestimating the model's performance in long, complex real-world conversations.
This work has significant implications for the development of real-time, low-latency multimodal assistants. By demonstrating that synthetic data can effectively train models for native audio-visual dialogue, it lowers the barrier to entry for developing such systems, which are currently hindered by the scarcity of high-quality, annotated real-world data. The focus on efficiency and style in the reward model aligns with user expectations for natural, concise interactions, making the research highly relevant to industry applications in smart devices and virtual assistants. The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $τ$ that bounds the decoder's lookahead. We show that $τ$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $τ$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($τ=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign
The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
The paper proposes a two-stage training recipe for streaming ASR using Delayed Streams Modeling (DSM). Stage 1 is standard Supervised Fine-Tuning (SFT) to reproduce the Voxtral Realtime baseline. Stage 2 introduces a Reinforcement Learning (RL) post-training phase using Group Relative Policy Optimization (GRPO). The core methodological contribution is the definition of a new latency metric, AWED (Aligned Word Emission Delay), which measures the time difference between the acoustic end of a word and its emission by the model. This metric is used to construct a joint reward function that balances transcription accuracy (WER) and latency. The use of GRPO is appropriate for the discrete, non-differentiable nature of emission timing decisions. The method effectively decouples the optimization of "what to say" and "when to say it" by leveraging the existing token prediction head, avoiding architectural changes.
The experiments are conducted on a large-scale dataset of ~175k hours of English speech. Evaluation is performed on eight diverse benchmarks from the HuggingFace Open ASR Leaderboard. The results demonstrate consistent improvements in the accuracy-latency Pareto frontier compared to the SFT baseline and the original Voxtral Realtime model. Specifically, the model achieves a 30.8% relative WER reduction at tight latency budgets (80ms) and maintains accuracy gains at higher latency budgets while reducing median AWED. Ablation studies on training duration and reward weight provide insight into the hyperparameter sensitivity, confirming the robustness of the chosen configuration.
The paper provides sufficient detail for reproducibility, including the specific base model (Voxtral-Mini-4B-Realtime), the use of LoRA for parameter efficiency, the data sources, and the evaluation protocol. The definition of AWED is clearly specified. However, the specific implementation details of the GRPO loop (e.g., group size, clipping epsilon) are not fully detailed in the text provided, though standard GRPO practices are implied. The reliance on a specific proprietary checkpoint (Voxtral) may limit reproducibility for those without access to that specific model version, but the method is generalizable.
The study is limited to English ASR. The evaluation relies on a specific base model (Voxtral), so the generalizability of the AWED+GRPO approach to other streaming ASR architectures (e.g., RNN-T based or other LALMs) is not tested. The latency metric AWED, while novel, is specific to the DSM paradigm and may not directly apply to other streaming approaches. The paper does not discuss computational overhead of the RL training phase compared to SFT.
This work addresses a critical practical issue in streaming ASR: the gap between nominal structural delay and perceived user latency. By providing a metric (AWED) and an optimization method (GRPO with joint reward) to close this gap, the paper offers a path to more responsive voice assistants and real-time transcription systems. The approach of using RL to optimize non-differentiable aspects of generation (timing) is a valuable technique that could be extended to other tasks involving sequential decision-making with latency constraints. The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns how, when, and whether the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.
Primary: Alibaba Group
All Institutions: Alibaba Token Foundry, Alibaba Group
The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
The paper proposes a comprehensive framework for real-time voice agents, structured around three layers: Think (foundation post-training), Act (agentic tool use), and Speak/Coordinate (conversational policy). The "Think" layer introduces a sophisticated post-training pipeline combining Core-Cocktail SFT with a novel "M$^2$-OPD" (Multimodality and Multi-Teacher On-Policy Distillation) strategy. This involves using a Text Teacher and a frozen Audio Reference to supervise student-generated trajectories, effectively transferring text-based reasoning capabilities to native audio models while preserving audio-specific nuances. The "Act" layer is particularly strong, introducing self-evolving executable environments for Group Relative Policy Optimization (GRPO). By using code agents to build, validate, and evolve tasks based on model performance (difficulty gating), the system learns robust tool use, state verification, and grounded progress communication. The "Speak/Coordinate" layer formalizes the decision-making process for when to speak, act, or remain silent, addressing critical issues in full-duplex interaction like background speech handling and turn-taking.
The evaluation is extensive, covering intelligence (audio reasoning, multilingual ASR, long-context), action (tool use, retrieval), interaction (persona, empathy, full-duplex behavior), and safety. Key results include a significant improvement in task success on the $\tau$-Voice benchmark (78.4% to 82.0%) and a dramatic reduction in response rate to background speech on Full-Duplex-Bench (73.0% to 13.0%), indicating much better conversational etiquette. The model also shows strong gains in multilingual audio understanding (BBA) and safety metrics. The inclusion of a "Voice Harness" prototype for persistent tasks adds depth to the system-level contribution.
The paper provides detailed descriptions of the training pipeline, data scales (approx. 1M hours), and evaluation protocols. However, as a technical report from a major industry lab, specific hyperparameters, exact dataset compositions, and code are not fully released, limiting independent reproduction. The reliance on in-house benchmarks (Long-AMC, WebSearch1K, VoiceChat) also restricts external verification.
The primary limitation is the lack of open-source code and data, making it difficult for the community to replicate the results. The evaluation relies heavily on in-house benchmarks and automatic judges (e.g., GPT-4o-mini, Qwen-Plus), which may introduce bias. The "Voice Harness" is presented as a prototype and is not fully integrated into the core model evaluation. Additionally, the paper focuses on a specific adaptation of $\tau$-Voice (half-duplex S2T) rather than the official full-duplex S2S protocol, which may limit direct comparability with other full-duplex systems.
This work significantly advances the state-of-the-art in real-time voice assistants by integrating robust agentic capabilities with natural conversational policies. The M$^2$-OPD technique and self-evolving environment for GRPO are valuable contributions to the broader field of multimodal LLM post-training. The focus on safety and reliable interaction (e.g., handling background speech, refusing unsupported requests) is crucial for the deployment of such systems in real-world scenarios. The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate "thinking" tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15\% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.
Primary: National Taiwan University
All Institutions: National Taiwan University, NVIDIA Research
The paper introduces a novel RL-based self-refinement framework for speech synthesis that enables a LALM to reason over its own audio output. By combining a generate-critique-refine pipeline with a specifically designed GRPO reward function that accounts for iterative improvement, the method achieves significant gains in instruction-following capability, demonstrating the viability of audio-token reasoning for enhancing generative quality.
The paper proposes a "Listen, Critique, and Refine" framework for instruction-following speech synthesis using Large Audio Language Models (LALMs). The core methodological contribution is extending the chain-of-thought/reasoning paradigm from text to audio tokens. The model generates a draft speech ($v_1$), generates a textual critique ($c$) by "listening" to $v_1$, and then generates a refined speech ($v_2$) conditioned on both. The training utilizes Group Relative Policy Optimization (GRPO) with a novel reward function. A key technical insight is the use of a non-linear transformation for the improvement term in the reward to prevent cancellation of the constant first-pass baseline within the GRPO group-relative advantage calculation. This is a sound and well-motivated technical detail that addresses a specific failure mode in applying GRPO to iterative refinement tasks.
The experiments are conducted on the InstructTTSEval benchmark, covering in-domain (DSD) and out-of-domain (APS, RP) tasks. The evaluation uses both objective metrics (CLSP score, WER) and subjective metrics (LALM-as-a-judge, human evaluation). The results show consistent improvements over zero-shot and RL one-hop baselines. The ablation studies effectively isolate the contributions of the non-linear reward shaping, speaker retrieval, and the critique mechanism. The human evaluation with 5 senior researchers adds credibility to the subjective gains. However, the training dataset is small (1000 examples from ParaSpeechCaps), which raises questions about the generalizability and robustness of the learned refinement behavior.
The authors provide a GitHub repository link, which is a positive step. The paper details the model architecture (Step-Audio-2-mini), hyperparameters, and implementation details (LoRA, vLLM, NCCL). However, the reliance on specific proprietary or less common models (Step-Audio-2-mini, CLSP) might limit reproducibility for the broader community compared to using more standard open-source components. The small training set size is a specific detail that is reproducible but may not be representative of large-scale applications.
The primary limitation is the small training dataset (1000 samples), which may lead to overfitting or limited generalization. The method relies on the base model's ability to generate meaningful critiques, which may not always be accurate or useful. The two-hop inference process doubles the computational cost compared to single-pass generation. The evaluation is limited to English speech. The reliance on CLSP for style adherence may not capture all nuances of complex instructions.
This work demonstrates the potential of self-refinement and reasoning in audio generation, a promising direction for improving the controllability and quality of LALMs. The technique of using non-linear reward shaping to handle iterative refinement under GRPO could be applicable to other multi-step generation tasks in other modalities. The framework provides a blueprint for integrating understanding and generation in a unified reasoning loop. The paper introduces a novel RL-based self-refinement framework for speech synthesis that enables a LALM to reason over its own audio output. By combining a generate-critique-refine pipeline with a specifically designed GRPO reward function that accounts for iterative improvement, the method achieves significant gains in instruction-following capability, demonstrating the viability of audio-token reasoning for enhancing generative quality.
Large audio-language models (LALMs) are increasingly used for a broader range of audio reasoning tasks. These models typically incorporate audio representations into a large language model (LLM) backbone to enable multimodal reasoning. Recent test-time reinforcement learning (TTRL) methods further improve LLM reasoning capability by leveraging unlabelled test data after pre-training. However, the importance of the perceptual capability of LALMs remains underexplored, particularly how much acoustic evidence is integrated and relied upon during reasoning, and how this contributes to final task performance. This gap limits the development of effective post-training methods like TTRL for audio reasoning. In this work, we first analyse how audio information is integrated and utilised during reasoning process. We quantify layer-wise perceptual reliance and show that stronger acoustic reliance is associated with higher accuracy and a larger performance gain attributable to the audio input. Building on this, we propose Perception-Grounded TTRL (PG-TTRL), which aligns label-free test-time optimisation with perceptually grounded reasoning, encouraging the model to structure its reasoning more strongly on the audio input. Experiments across LALMs and benchmarks show that PG-TTRL consistently improves reasoning performance over both the base models and standard TTRL, showing the value of perceptual-grounding optimisation for test-time audio reasoning.
Primary: University of Maryland, College Park
All Institutions: University of Maryland, College Park, University of Illinois Urbana-Champaign, University of Washington
The paper presents a rigorous analysis of perceptual reliance in LALMs and proposes PG-TTRL, a novel test-time reinforcement learning method that aligns policy optimization with acoustic grounding, resulting in consistent performance gains over standard TTRL and base models.
The paper introduces a novel diagnostic framework for Large Audio-Language Models (LALMs) that quantifies "perceptual reliance" by comparing hidden representations under full audio access versus masked audio attention. This analysis reveals that acoustic evidence integration peaks in intermediate layers and correlates with task accuracy. Building on this, the authors propose PG-TTRL (Perception-Grounded Test-Time Reinforcement Learning), which modifies the Group Relative Policy Optimization (GRPO) advantage function. Instead of relying solely on majority vote correctness, PG-TTRL computes a trajectory-level grounding score based on the log-probability gap between original and masked audio conditions. This score is used to derive a reliability weight that scales the advantage, effectively penalizing updates that rely on language priors rather than acoustic evidence. The method is theoretically sound, addressing a critical flaw in standard TTRL where models may reinforce linguistic shortcuts. The integration of causal intervention (attention masking) into the reward signal is a creative and rigorous approach to grounding multimodal reasoning.
Experiments are conducted on two state-of-the-art LALMs (Qwen2.5-Omni-3B and 7B) across two major benchmarks (MMAR and MMAU). The results show consistent improvements over both base models and standard TTRL, with gains up to 4.9 points on MMAU. The analysis of Pass@k metrics is particularly compelling, demonstrating that PG-TTRL preserves diverse correct trajectories better than standard TTRL, which tends to collapse to incorrect majority votes on hard instances. The ablation study on layer-wise changes confirms that the method specifically enhances late-stage reasoning utilization of audio, validating the hypothesis. However, the evaluation is limited to multiple-choice tasks, and the sample size for correlation analysis is modest.
The paper provides detailed hyperparameters, including LoRA ranks, learning rates, and specific bounds for the reliability weight. The algorithm is clearly defined, and the reproducibility checklist is filled out, indicating that code will be released. The use of standard benchmarks and open-source models (Qwen) enhances reproducibility. However, the specific implementation of the attention masking for the grounding score requires careful handling of the model's internal attention mechanisms, which may be non-trivial to replicate without the provided code.
The primary limitation is the restriction to multiple-choice audio reasoning tasks; the method's applicability to open-ended generation or other modalities is not explored. The grounding score relies on teacher-forcing passes, which may not perfectly reflect the autoregressive generation process. Additionally, the method adds computational overhead due to the extra forward passes required for grounding score calculation. The correlation analysis excludes small-sample tasks, which may limit the generalizability of the findings.
This work addresses a fundamental issue in multimodal AI: the tendency of models to ignore perceptual inputs in favor of language priors. By providing a mechanism to explicitly reward perceptual grounding, the paper offers a pathway to more robust and trustworthy multimodal systems. The insights into layer-wise perceptual reliance are valuable for the broader community developing multimodal architectures. The method could be extended to other modalities (vision, video) and tasks, potentially improving the reliability of AI systems in safety-critical applications where accurate perception is crucial. The paper presents a rigorous analysis of perceptual reliance in LALMs and proposes PG-TTRL, a novel test-time reinforcement learning method that aligns policy optimization with acoustic grounding, resulting in consistent performance gains over standard TTRL and base models.
Constraint-following music generation asks a score to satisfy several user-specified properties at once, each checkable programmatically (key, meter, length, range, final note, rhythm, motion and form), yet no existing benchmark isolates this capability. We construct MusicConstraintBench, 2,180 items over eight constraint families, on which current models fail once a few constraints are combined. The natural remedy is reinforcement learning with these verifiers as reward, yet we observe that a reward paid only when every property holds leaves most training groups without a learning signal: over the first 50 updates, 0.550 of rollout groups score identically and receive no gradient, even though a failing score typically misses only one requested property. Under the joint criterion, rollouts for a prompt tend to fail together, so a binary reward cannot separate a nearly correct score from a malformed one. We therefore introduce MusicRLVR, which pays graded per-property credit behind a hard validation gate that rejects malformed outputs, plus a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain fine-tuning. On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints and leads every zero-shot baseline including Llama-3.1-70B at 0.380. It also generalises to property combinations unseen in training and to out-of-range parameter values, showing that verifiable rewards need not presuppose a target output.
Primary: Anonymous (Double-Blind Review)
All Institutions: Anonymous
The paper introduces a verifier-driven reinforcement learning framework and benchmark for constraint-following symbolic music generation, demonstrating that graded, deterministic rewards can significantly improve multi-constraint compliance in large language models without requiring music-domain supervised fine-tuning.
The paper proposes MusicRLVR, a reinforcement learning framework that uses deterministic, programmatic verifiers as rewards for symbolic music generation. The core methodological contribution is the design of a graded reward function that combines a hard validity gate (rejecting malformed ABC notation) with per-family partial credit and a joint-satisfaction bonus. This addresses the "sparse reward" problem in GRPO where binary success/failure signals lead to zero-variance groups and no gradient updates. The approach is technically sound, leveraging the fact that musical constraints (key, meter, form, etc.) are programmatically checkable. The use of GRPO (Group Relative Policy Optimization) is appropriate for this setting, and the ablation studies effectively demonstrate the necessity of the graded credit component over binary rewards.
The authors introduce MusicConstraintBench, a new benchmark of 2,180 items covering eight constraint families. The experiments are rigorous, comparing the proposed method against strong zero-shot baselines (including Llama-3.1-70B) and supervised fine-tuning (SFT) baselines. The results show significant improvements in constraint compliance, particularly on multi-constraint tasks where baselines fail. The evaluation includes tests for compositional generalization (unseen constraint combinations) and parameter extrapolation, which are critical for assessing the robustness of the learned policy. The statistical significance is supported by McNemar tests and Holm correction.
The paper provides extensive details on the training setup, hyperparameters, and data construction. It explicitly states that code, benchmark suites, and model outputs will be released under an MIT license. The reproducibility statement is detailed, mentioning specific scripts for asset generation and evaluation. The use of standard tools like vLLM and ms-swift enhances reproducibility.
The primary limitation is the domain specificity; the method is tailored to symbolic music in ABC notation and may not directly transfer to other domains without significant adaptation. Additionally, the paper acknowledges that compliance scores do not measure perceptual musical quality, which is a significant gap for practical applications. The reliance on a specific corpus (IrishMAN) for SFT data might introduce stylistic biases, although the RL phase is designed to be independent of this.
This work demonstrates that RLVR (Reinforcement Learning with Verifiable Rewards) can be applied to open-ended generation tasks where the output space is large but the properties of interest are verifiable. This has implications for other structured generation tasks (e.g., code, math, structured data) where deterministic checks are possible. It provides a template for building benchmarks and reward functions for property-constrained generation. The paper introduces a verifier-driven reinforcement learning framework and benchmark for constraint-following symbolic music generation, demonstrating that graded, deterministic rewards can significantly improve multi-constraint compliance in large language models without requiring music-domain supervised fine-tuning.
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
Primary: University of Maryland
All Institutions: Adobe Research, University of Maryland, OpenAI
[One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
The paper proposes a structured taxonomy of 22 paralinguistic and acoustic characteristics, which is a significant conceptual contribution to the field. The methodology relies heavily on a data simulation engine that augments clean speech with Room Impulse Responses (RIRs) and noise to create controlled acoustic environments. This is a standard but effective approach for acoustic tasks. The core innovation lies in the two-stage curriculum learning strategy: Stage 1 focuses on atomic single-attribute QA pairs to build foundational knowledge, while Stage 2 uses LLM-generated multi-attribute QA pairs to foster joint reasoning. The use of Qwen2.5-7B for generating training data and Mistral-Small-3.2 for generating benchmark questions to avoid self-referential bias is a thoughtful methodological choice. However, the reliance on synthetic data for acoustic characteristics and existing datasets for speaker traits limits the generalizability to real-world, uncontrolled acoustic scenes. The model is fine-tuned from Qwen2-Audio-7B-Instruct using LoRA, which is a standard and efficient approach.
The experimental setup is robust, introducing ParA-Bench, a 6,000-question multiple-choice benchmark. The evaluation compares ParA-LLM against state-of-the-art models like GPT-4o-Audio, Qwen2-Audio, and specialized reasoning models. The results show a 7.5% improvement over GPT-4o-Audio on the proposed benchmark, which is a strong result. The ablation study on the curriculum learning shows consistent gains, validating the training strategy. The inclusion of downstream applications, such as Text-to-Impulse Response (Text2IR) generation, demonstrates the practical utility of the learned representations. However, the evaluation is primarily limited to the proposed benchmark and a few standard audio benchmarks (MMAU-Pro, MMAR). There is a lack of evaluation on other established paralinguistic benchmarks (e.g., IEMOCAP for emotion, or specific speaker verification datasets) to fully validate the model's capabilities across different paralinguistic tasks.
The paper provides detailed information on the data sources (EARS, Emilia, Expresso, VoxCeleb, MIT IR Survey, etc.), the simulation parameters, and the training hyperparameters (LoRA rank, learning rates, batch sizes). The release of the dataset, model, and benchmark significantly enhances reproducibility. The use of specific models for data generation (Qwen2.5-7B, Mistral-Small-3.2) is clearly stated. However, the exact prompts used for LLM-based QA generation are not fully detailed in the text, which could be a minor barrier to exact reproduction of the data pipeline.
The primary limitation is the reliance on synthetic acoustic data for the acoustic characteristics, which may not fully capture the complexity and variability of real-world acoustic environments. The speaker and speech characteristics are derived from existing datasets, which may have inherent biases or limited diversity. The model is based on a 7B parameter LLM, which may limit its scalability to more complex reasoning tasks compared to larger models. The benchmark, while novel, is relatively small (6K questions) and may not fully represent the breadth of paralinguistic understanding. The paper does not extensively discuss the computational cost of the data generation pipeline or the potential biases introduced by the LLMs used for data augmentation.
This work has significant potential impact on the field of audio AI by addressing a critical gap in paralinguistic understanding. The release of a large-scale dataset and a comprehensive benchmark will facilitate future research in this area. The demonstrated downstream applications, such as TTS control and acoustic space captioning, highlight the practical value of the model. The framework could be extended to other modalities or languages, and the curriculum learning approach could be applied to other multi-attribute understanding tasks. The work also contributes to the broader goal of building more human-like audio AI systems that can understand not just what is said, but how it is said. [One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Existing benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evaluate continuous and discrete representations across speech, sound and music. Utilizing our UniARC framework with dual evaluation strategies across model scales from SmolLM2-135M to Llama-3-8B, we analyze the dynamic relationships of data volume, model capacity, and computational efficiency. Our results reveal the pivotal role of semantic constraints in tokenization for audio understanding and demonstrate that scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks. These findings offer practical guidance for balancing semantic density, fidelity, and efficiency in future LALMs.
Primary: Tsinghua University
All Institutions: Tsinghua University
The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
The paper proposes the UniARC framework, a systematic evaluation methodology for comparing continuous (SSL-based) and discrete (clustering/codec-based) audio representations within Large Audio Language Models (LALMs). The methodology is sound, utilizing two distinct evaluation strategies: parameter-efficient fine-tuning (LoRA) on smaller backbones (SmolLM2) and frozen-backbone probing on larger backbones (Llama-3). This dual approach effectively isolates the impact of representation quality from the capacity of the language model. The choice of encoders is comprehensive, covering major SSL models (HuBERT, WavLM, Whisper) and discrete tokenizers (K-means, DAC, WavTokenizer, SpeechTokenizer). The standardization of feature processing (projectors, temporal concatenation) ensures fair comparison.
The experiments span three domains (speech, sound, music) and multiple tasks (ASR, emotion, captioning, classification). The results provide clear insights: semantic alignment is more critical than raw fidelity for understanding tasks. The finding that scaling the LLM backbone does not compensate for poor audio representation quality is a significant and counter-intuitive result that adds value. The analysis of data diversity and computational efficiency further strengthens the contribution. However, the sample size of encoders is limited, and the specific hyperparameters for the K-means clustering (1000 centroids) might not be optimal for all scales, though the authors justify this choice.
The paper provides sufficient details for reproduction, including model names, dataset names, and general training strategies (LoRA rank, learning rates). However, specific details on the projector architecture (beyond "two-layer MLP") and the exact prompt templates used for instruction tuning are not fully detailed in the provided text. The reliance on open-source models (SmolLM2, Llama-3, XARES-LLM base) aids reproducibility.
The study is limited to specific model scales (up to 8B) and a specific set of encoders. It does not explore hybrid approaches in depth, only comparing pure continuous vs. pure discrete. The "inverse scaling" phenomenon observed might be confounded by the simple MLP projector, as the authors acknowledge. The lack of ablation studies on the projector design is a minor weakness.
The findings have practical implications for the design of future LALMs, suggesting that investment in semantically rich audio encoders is more valuable than simply scaling up the LLM backbone for understanding tasks. This guides resource allocation in the field. The emphasis on multi-domain pre-training for encoders is also a valuable takeaway. The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Primary: Alibaba Group
All Institutions: The Chinese University of Hong Kong, Alibaba Group, Shanghai Jiao Tong University, Shanghai Innovation Institute, Zhejiang University
The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
The paper introduces a coherent framework for "native" audio-visual dialogue, addressing the critical gap in data availability for models that process raw audio/video without ASR intermediaries. The core methodological contribution is OmniVChat-Studio, a multi-agent synthesis engine. The design is robust, utilizing distinct agents (Director, Renderer, Reviewer, Validator) to handle the complex pipeline of script generation, video rendering, and quality verification. The separation of "Fixed" and "Flexible" modules allows for extensibility to new subcategories without re-engineering the core pipeline. The proposed OmniVChat-RL reward design is particularly strong, moving beyond simple correctness to incorporate efficiency (conciseness) and style (naturalness), which are crucial for conversational agents. The use of GSPO (Group Sequence Policy Optimization) with LoRA is a practical and effective training strategy for large multimodal models.
The experimental setup is rigorous, featuring a synthetic benchmark (OmniVChat-Bench) and a human-recorded transfer probe (OmniVChat-Bench-Human). The evaluation metrics are well-defined, using LLM-based grading with tiered rubrics to handle the open-ended nature of dialogue. The results show significant improvements in both correctness and efficiency (reply length reduction) when using the proposed RL method. The comparison against state-of-the-art models (Gemini, Qwen, etc.) provides a strong baseline. The ablation studies effectively isolate the contributions of the efficiency and style rewards. The transfer from synthetic to human data is a key validation point, and the results support the claim that synthetic data can effectively train models for real-world scenarios.
The paper provides detailed descriptions of the agent roles, sampling strategies, and reward functions. The code and data are promised to be released, which is standard for such contributions. The specific prompts for the Director and Reviewer agents are referenced in appendices, which aids reproducibility. However, the reliance on specific proprietary video generation models (implied by the "Renderer" agent) and LLMs for grading may limit full reproducibility for external groups without access to these specific tools.
The primary limitation is the reliance on synthetic data for training and evaluation. While the paper argues for the validity of this approach, the "reality gap" between synthesized and real-world audio-visual noise, lighting, and human behavior remains a concern, even with the human-recorded probe. The evaluation relies heavily on LLM-as-a-judge, which can introduce biases or inconsistencies, although the paper attempts to mitigate this with detailed rubrics and agreement checks. The multi-turn evaluation is limited to single-turn probes in the human dataset, potentially underestimating the model's performance in long, complex real-world conversations.
This work has significant implications for the development of real-time, low-latency multimodal assistants. By demonstrating that synthetic data can effectively train models for native audio-visual dialogue, it lowers the barrier to entry for developing such systems, which are currently hindered by the scarcity of high-quality, annotated real-world data. The focus on efficiency and style in the reward model aligns with user expectations for natural, concise interactions, making the research highly relevant to industry applications in smart devices and virtual assistants. The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: \texttt{
Primary: Meta Reality Labs
All Institutions: Meta Reality Labs
The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
The paper proposes Interrupt and Silent Modeling (ISM), a model-agnostic framework that enables proactive audio assistance in AudioLLMs. The core method involves extending the vocabulary of a standard AudioLLM (Qwen2-Audio-7B) with two special tokens,
The experiments are conducted on ESC-50 (training and evaluation) and Epic-Sounds (zero-shot evaluation). The results on ESC-50 are exceptionally high (99.6% F1), which is somewhat suspiciously perfect for a real-world audio task, suggesting potential overfitting or a very easy evaluation setup (e.g., clean, isolated clips). The zero-shot transfer to Epic-Sounds is a strong point, showing the model generalizes to noisy, egocentric kitchen audio without domain-specific training. The streaming evaluation confirms real-time viability with 3.5s latency. However, the lack of comparison with other proactive or streaming audio models (if any exist) and the limited number of baselines (Zero-Shot, Reactive SFT) weakens the comparative analysis. The "perfect" de-duplication recall is also a red flag that warrants scrutiny of the evaluation protocol.
The paper provides sufficient details on the architecture (Qwen2-Audio-7B, LoRA rank 8), training procedure (two-stage SFT), and data construction for the four states. However, the exact prompts used for the "watch-out intent" and the specific construction of the streaming evaluation set are not fully detailed in the provided text. The code and data are not explicitly linked, which hinders immediate reproducibility.
The primary limitation is the computational cost of using a 7B LLM for a binary decision task, which may be prohibitive for battery-powered wearable devices despite the latency claim. The near-perfect scores on ESC-50 may not reflect real-world performance in highly complex, overlapping acoustic scenes. The paper focuses only on explicit intents, leaving implicit and semantic assistance for future work. The lack of user studies with DHH participants limits the validation of the system's practical utility.
This work has significant potential for assistive technology, particularly for Deaf and Hard of Hearing (DHH) users who rely on sound awareness. By enabling proactive, intent-based alerts with de-duplication, it addresses key pain points in current wearable sound awareness systems (notification fatigue, lack of personalization). The model-agnostic nature of ISM makes it a portable capability that can be integrated into future AudioLLMs, potentially driving a shift from reactive to proactive audio AI. The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
The success of Large Audio Language Models has driven the development of massive multimodal networks exceeding billions of parameters. However, the demand for privacy-preserving, low-latency processing has shifted focus toward Small Audio Language Models (SALMs) capable of on-device execution. In this paper, we introduce Samsone, a family of SALMs designed for edge computing. Our core model, Samsone-134M, establishes a new state-of-the-art for its size class across multiple benchmarks. We further explore the scaling laws of SALMs by introducing Samsone-99M and Samsone-356M. Despite their compact footprint, the Samsone family delivers performance competitive with models orders of magnitude larger. To foster open research and reproducibility, we train Samsone on publicly available data. We release the training code, model weights, mobile-optimized checkpoints and provide an open-source Android application to demonstrate real-time on-device inference of Samsone.
Primary: Samsung R&D Institute Poland
All Institutions: Samsung R&D Institute Poland, AGH University of Kraków
The paper introduces Samsone, a family of small audio language models (SALMs) that achieve state-of-the-art performance for their size class by applying vocabulary reduction and depth pruning to standard encoder-decoder architectures. While the architectural components are not novel, the rigorous engineering of these models for on-device inference, combined with strong benchmark results that challenge the necessity of massive parameter counts for audio understanding, represents a significant practical contribution to the field of efficient multimodal AI.
The methodology relies on a standard "encoder-projector-decoder" architecture, which is not novel in itself. The core technical contribution lies in the systematic application of size optimization techniques—specifically vocabulary reduction and depth pruning—to existing open-source components (Whisper-Tiny encoder and SmolLM2 backbone). The approach is pragmatic rather than architecturally groundbreaking. The use of a SEP token to handle multiple audio inputs is a sensible design choice for a general-purpose audio LM, but it is not a significant theoretical advance. The training procedure is straightforward single-stage fine-tuning, lacking the complex multi-stage pre-training or alignment strategies seen in larger LALMs.
The experimental evaluation is the strongest aspect of the paper. The authors provide a comprehensive comparison against both Large Audio Language Models (LALMs) and existing Small Audio Language Models (SALMs) on rigorous benchmarks like MMAU and MMAU-Pro. The results are compelling: Samsone-134M significantly outperforms previous SALMs (Mellow, Pengi) and approaches the performance of models 60x larger (Qwen2-Audio). The inclusion of on-device latency benchmarks on a specific hardware (Samsung Galaxy S25 Ultra) adds practical value, validating the "on-device" claim. However, the ablation study is somewhat limited, testing only a few alternative components without deep diving into the impact of specific training data mixes or the exact trade-offs of the pruning strategies beyond the final performance numbers.
High. The authors explicitly state they release training code, model weights, and an Android application. The use of publicly available datasets (ReasonAQA, AudioSkillsXL) and open-source base models (Whisper, SmolLM2) ensures that the work can be easily reproduced by the community. The detailed description of the training setup (optimizer, learning rate, epochs) further supports reproducibility.
The paper acknowledges that the models suffer from a loss of general-purpose linguistic capabilities due to extensive AQA fine-tuning. This is a common issue in domain-specific fine-tuning but limits the model's versatility. Additionally, the evaluation is heavily skewed towards multiple-choice and short-answer tasks (AQA); performance on open-ended generation or complex reasoning tasks beyond the benchmark scope is less clear. The lack of hardware-specific optimizations (NPU/GPU) means the reported latency may not represent the best possible performance on modern mobile devices.
This work is highly relevant to the growing field of edge AI and privacy-preserving audio processing. By demonstrating that sub-100M parameter models can achieve competitive performance with billion-parameter models, it lowers the barrier to entry for deploying advanced audio understanding on consumer devices. This has implications for real-time applications in healthcare, smart home assistants, and mobile interfaces where data privacy and latency are critical. The paper introduces Samsone, a family of small audio language models (SALMs) that achieve state-of-the-art performance for their size class by applying vocabulary reduction and depth pruning to standard encoder-decoder architectures. While the architectural components are not novel, the rigorous engineering of these models for on-device inference, combined with strong benchmark results that challenge the necessity of massive parameter counts for audio understanding, represents a significant practical contribution to the field of efficient multimodal AI.
Computational models of auditory physiology commonly target specific responses or stages of the auditory pathway, limiting their ability to integrate findings across experimental paradigms and neural timescales. We present a foundation model of human auditory electrophysiology: a causal neural network trained to map binaural acoustic waveforms directly to high-sample-rate EEG. The model was trained on approximately 250 hours of EEG data from 92 subjects, with varied electrode montages and stimuli spanning tonebursts, speech, and music. We tested whether the model recovered effects of stimulus rate, frequency, and presentation method on auditory brainstem responses (ABRs); subcortical and cortical temporal response functions (TRFs) to continuous speech; and the click-evoked binaural interaction component (BIC). Predicted ABRs and TRFs reproduced established response morphology and stimulus-dependent effects, with model-grand-average correlations falling within the corresponding subject-level human distributions. The model-predicted BIC metrics closely resembled the values reported in the literature. These findings demonstrate that a single audio-to-EEG model can capture auditory physiology across paradigms and timescales, supporting future in silico experimentation and hearing technology applications.
Primary: University of Michigan
All Institutions: University of Michigan, Kresge Hearing Research Institute, Department of Otolaryngology--Head and Neck Surgery
The paper presents a causal deep neural network that maps binaural audio to high-sample-rate EEG, demonstrating the ability to reproduce established auditory physiological phenomena across multiple paradigms. The technical contribution is significant in its integration of diverse stimuli and subjects into a single model, offering a promising direction for in silico experimentation in auditory neuroscience and hearing technology development.
The authors propose a "foundation model" for auditory electrophysiology, utilizing a causal encoder-decoder architecture (WaveNet-based) to map binaural acoustic waveforms directly to high-sample-rate EEG. The methodology is robust, employing a 24-band filterbank, log-compression, and a 9-layer causal WaveNet encoder with dilated convolutions. A key technical contribution is the use of a Short-Time Fourier Transform (STFT) domain loss function that penalizes phase and amplitude errors, which is more suitable for EEG signal reconstruction than standard MSE. The inclusion of a parallel linear pathway to model stimulus-locked electromagnetic artifacts is a sophisticated engineering choice that improves training stability. The model is conditioned on subject identity and demographics, allowing for potential personalization, though the primary evaluation uses a "default" subject embedding.
The evaluation is comprehensive, testing the model on three distinct paradigms: parallel auditory brainstem responses (pABR), speech temporal response functions (TRFs), and binaural interaction components (BIC). The dataset is substantial (250 hours, 92 subjects), covering diverse stimuli (speech, music, tonebursts) and electrode montages. The results show that the model reproduces established physiological phenomena, such as frequency-dependent ABR morphology and rate-dependent latency/amplitude changes. Statistical comparisons (Crawford-Howell tests) indicate that model predictions fall within the distribution of human subject-level correlations, validating the model's biological plausibility. However, the lack of a direct comparison with state-of-the-art non-foundation models (e.g., specific ABR predictors or TRF estimators) limits the clarity of its relative performance advantage.
The paper provides detailed architectural specifications (kernel sizes, dilations, filterbank parameters) and training hyperparameters (learning rate, batch size, loss function details). The description of the artifact pathway and the STFT loss is sufficient for replication. However, the code and pre-trained weights are not explicitly linked in the provided text, and the specific implementation of the "default subject" zero-shot mechanism could benefit from further clarification. The data curation process (clock drift correction, resampling) is described, which is crucial for high-sample-rate EEG work.
The model is trained on a relatively young cohort (mean age 22.6 years), limiting generalizability to older populations or those with hearing loss. The evaluation relies heavily on correlation with grand averages and published statistics rather than direct prediction accuracy metrics (like SNR or RMSE) against held-out individual subjects in a standardized way. The "foundation model" claim is somewhat strong given the single-modality input (audio) and output (EEG) and the specific domain of auditory physiology; it is more accurately described as a specialized domain-specific model. The paper does not discuss computational cost or real-time inference capabilities, which are critical for the proposed hearing aid applications.
This work has significant potential for advancing computational auditory neuroscience by providing a unified framework for simulating brain responses to sound. It could accelerate the development of hearing technologies by enabling in silico testing of hearing aid algorithms. The causal nature of the model makes it suitable for real-time applications. However, the impact is currently limited to the auditory domain and specific electrophysiological signals, and broader impact would require validation on larger, more diverse datasets and integration with other neural signals. The paper presents a causal deep neural network that maps binaural audio to high-sample-rate EEG, demonstrating the ability to reproduce established auditory physiological phenomena across multiple paradigms. The technical contribution is significant in its integration of diverse stimuli and subjects into a single model, offering a promising direction for in silico experimentation in auditory neuroscience and hearing technology development.
Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy which traditional MOS predictors fail to capture. While proprietary Large Language Models (LLMs) like Gemini can evaluate these aspects, they are too costly for massive inference and reinforcement learning feedback. To address this, we first introduce Live-ProsodyJudge (LPJ), a cost-effective pairwise evaluator distilled from Gemini into Qwen3-Omni. However, we identify a critical flaw in standard multi-dimensional evaluation: verdict coupling. The judge tends to lazily align all individual dimension scores with its overall preference, collapsing a rich multi-dimensional rubric into a single preference bit. To resolve this, we further propose Decoupled-Live-ProsodyJudge (D-LPJ). D-LPJ eliminates the overall verdict target to prevent blind following, masks uncertain pair-dimensions during Supervised Fine-Tuning(SFT), and introduces a novel span-local GRPO strategy that applies normalized advantages strictly to their corresponding rationale spans. Evaluated on highly curated human-annotated test sets, 10 sample balanced-order LPJ achieves higher point accuracy than a single Gemini call, while D-LPJ successfully produces independent,decoupled dimension judgments. Furthermore, in a Best-of-8 TTS candidate selection tournament, the LPJ-selected utterance falls within the human top-3 in 85.29% of high-confidence cases, demonstrating its efficacy for fine-grained TTS preference optimization.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, TaoLive-AIGC Team, Taobao & Tmall Group of Alibaba
The paper introduces a decoupled multi-dimensional prosody judge for live-streaming TTS, mitigating verdict coupling through span-local GRPO and achieving higher accuracy than its proprietary teacher model. This work provides a robust, cost-effective framework for fine-grained speech evaluation, enabling scalable preference optimization and offering a novel solution to the problem of correlated judgments in multi-dimensional LLM-based evaluators.
The paper proposes a two-stage approach to evaluating live-streaming TTS. First, it distills a proprietary LLM (Gemini) into an open-weight model (Qwen3-Omni) to create a cost-effective pairwise judge (LPJ). The core methodological contribution is the identification and mitigation of "verdict coupling," where judges lazily align all dimension scores with a single overall preference. To solve this, the authors propose D-LPJ, which removes the overall verdict target, masks uncertain dimensions during SFT, and introduces a "span-local GRPO" strategy. This reinforcement learning technique applies normalized advantages strictly to the rationale spans of specific dimensions, preventing cross-dimension reward cancellation. The methodology is sound, leveraging standard distillation techniques but applying them to a specific failure mode in multi-dimensional evaluation. The span-local GRPO is a clever application of RLHF principles to structured output generation.
The experiments are rigorous for an industry-adjacent paper. The authors construct a curated dataset of 1,043 human-annotated pairs, controlling for lexical confounding by using matched transcripts. They evaluate on multiple test sets (T1-T5) covering different TTS models and human recordings. Key results show that the distilled model (LPJ) outperforms a single call of the teacher model (Gemini) in point accuracy, which is a strong claim. The decoupled model (D-LPJ) successfully produces varied dimension judgments, with 86.10% agreement on high-confidence sets. The Best-of-8 selection experiment demonstrates practical utility, with the judge's top choice falling within the human top-3 in 85.29% of cases. The inclusion of position bias diagnostics and ablation studies (e.g., removing stage-1 SFT) adds depth to the evaluation.
Reproducibility is moderate. The paper provides detailed hyperparameters (LoRA ranks, learning rates, batch sizes) and describes the data construction process. However, the reliance on proprietary models (Gemini for distillation, Qwen3-Omni as base) and internal datasets (TaoLive data) limits full external reproducibility. The code is not explicitly linked, and the specific "span-local GRPO" implementation details, while described mathematically, would require significant effort to replicate without the exact codebase. The use of internal human annotations for the test sets is a strength for validity but a barrier for independent verification.
The primary limitation is the dependence on proprietary teacher models for distillation, which may not be accessible to all researchers. The evaluation is heavily focused on live-streaming TTS, which is a specific niche; generalizability to other speech domains (e.g., audiobooks, conversational agents) is not tested. The "span-local GRPO" is a novel idea but its computational cost and stability compared to standard GRPO are not deeply analyzed. The paper acknowledges that the decoupled model is currently limited to core dimensions (C1-C4) and does not fully address the sparse conditional dimensions (L1-L3) in the decoupled setting.
This work has significant impact on the field of speech synthesis evaluation. By providing a scalable, cost-effective alternative to expensive LLM-based judges, it enables more frequent and fine-grained evaluation of TTS systems, which is crucial for RLHF-based optimization. The concept of "verdict coupling" and its mitigation via span-local rewards could be applied to other multi-dimensional evaluation tasks in NLP and audio. The Best-of-N selection framework demonstrated here is directly applicable to production TTS pipelines. The paper introduces a decoupled multi-dimensional prosody judge for live-streaming TTS, mitigating verdict coupling through span-local GRPO and achieving higher accuracy than its proprietary teacher model. This work provides a robust, cost-effective framework for fine-grained speech evaluation, enabling scalable preference optimization and offering a novel solution to the problem of correlated judgments in multi-dimensional LLM-based evaluators.
Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $τ$ that bounds the decoder's lookahead. We show that $τ$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $τ$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($τ=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign
The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
The paper proposes a two-stage training recipe for streaming ASR using Delayed Streams Modeling (DSM). Stage 1 is standard Supervised Fine-Tuning (SFT) to reproduce the Voxtral Realtime baseline. Stage 2 introduces a Reinforcement Learning (RL) post-training phase using Group Relative Policy Optimization (GRPO). The core methodological contribution is the definition of a new latency metric, AWED (Aligned Word Emission Delay), which measures the time difference between the acoustic end of a word and its emission by the model. This metric is used to construct a joint reward function that balances transcription accuracy (WER) and latency. The use of GRPO is appropriate for the discrete, non-differentiable nature of emission timing decisions. The method effectively decouples the optimization of "what to say" and "when to say it" by leveraging the existing token prediction head, avoiding architectural changes.
The experiments are conducted on a large-scale dataset of ~175k hours of English speech. Evaluation is performed on eight diverse benchmarks from the HuggingFace Open ASR Leaderboard. The results demonstrate consistent improvements in the accuracy-latency Pareto frontier compared to the SFT baseline and the original Voxtral Realtime model. Specifically, the model achieves a 30.8% relative WER reduction at tight latency budgets (80ms) and maintains accuracy gains at higher latency budgets while reducing median AWED. Ablation studies on training duration and reward weight provide insight into the hyperparameter sensitivity, confirming the robustness of the chosen configuration.
The paper provides sufficient detail for reproducibility, including the specific base model (Voxtral-Mini-4B-Realtime), the use of LoRA for parameter efficiency, the data sources, and the evaluation protocol. The definition of AWED is clearly specified. However, the specific implementation details of the GRPO loop (e.g., group size, clipping epsilon) are not fully detailed in the text provided, though standard GRPO practices are implied. The reliance on a specific proprietary checkpoint (Voxtral) may limit reproducibility for those without access to that specific model version, but the method is generalizable.
The study is limited to English ASR. The evaluation relies on a specific base model (Voxtral), so the generalizability of the AWED+GRPO approach to other streaming ASR architectures (e.g., RNN-T based or other LALMs) is not tested. The latency metric AWED, while novel, is specific to the DSM paradigm and may not directly apply to other streaming approaches. The paper does not discuss computational overhead of the RL training phase compared to SFT.
This work addresses a critical practical issue in streaming ASR: the gap between nominal structural delay and perceived user latency. By providing a metric (AWED) and an optimization method (GRPO with joint reward) to close this gap, the paper offers a path to more responsive voice assistants and real-time transcription systems. The approach of using RL to optimize non-differentiable aspects of generation (timing) is a valuable technique that could be extended to other tasks involving sequential decision-making with latency constraints. The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.
Primary: University of California Los Angeles
All Institutions: University of California Los Angeles
Proposes a two-stage adapter-based fine-tuning method that effectively adapts the speech encoder in Speech-LLMs to domain-shifted data. The paper demonstrates that targeted "awakening" of the encoder via lightweight adapters, followed by joint fine-tuning, significantly outperforms standard fine-tuning strategies, achieving new state-of-the-art results on child and dialectal speech benchmarks while preserving pre-trained knowledge.
The paper proposes EAVA, a two-stage fine-tuning strategy for Speech-LLMs. The core insight is that standard fine-tuning often fails to adequately adapt the speech encoder to new acoustic domains because the LLM dominates the gradient flow. Stage 1 ("Encoder Awakening") freezes the entire pre-trained model and trains only lightweight adapters (Residual Adapters) inserted into the encoder layers. Stage 2 jointly fine-tunes the encoder, adapters, projector, and LLM LoRA. The methodology is sound, leveraging the parameter efficiency of adapters to preserve pre-trained knowledge while injecting domain-specific acoustic features. The use of CKA analysis to visualize layer-wise representation shifts provides a strong theoretical grounding for why this approach works (preserving lower layers, adapting upper layers).
The experiments are comprehensive, covering three distinct domain-shifted datasets (OGI Kids, MyST, CORAAL) and two different Speech-LLM backbones (Canary-Qwen, Phi-4-Multimodal). The paper reports state-of-the-art results, outperforming vanilla fine-tuning, multi-stage alignment, and previous SOTA methods. Ablation studies effectively isolate the contribution of the two-stage process and the choice of adapter architecture. The inclusion of cross-dataset transfer analysis and loss function variations (CTC vs. CE) adds significant depth to the evaluation.
The paper provides a public GitHub repository link. It specifies model architectures, dataset splits, hyperparameters (learning rates, adapter dimensions), and training schedules. The use of open-source backbones (Canary-Qwen, Phi-4) and standard datasets enhances reproducibility.
The method relies on the availability of a pre-trained Speech-LLM backbone. The gains, while consistent, are moderate in some cases (e.g., MyST). The paper does not extensively explore the computational overhead of the two-stage training compared to single-stage fine-tuning, though the parameter efficiency is highlighted. The "awakening" stage requires a separate training phase, which may be a barrier for rapid deployment scenarios.
This work addresses a critical bottleneck in deploying Speech-LLMs in real-world, diverse acoustic environments. By providing a parameter-efficient and effective domain adaptation method, it lowers the barrier for adapting large speech models to specific populations (children, dialect speakers) or noisy environments. The insights into encoder representation shifts via CKA are valuable for the broader community working on multimodal model adaptation. Proposes a two-stage adapter-based fine-tuning method that effectively adapts the speech encoder in Speech-LLMs to domain-shifted data. The paper demonstrates that targeted "awakening" of the encoder via lightweight adapters, followed by joint fine-tuning, significantly outperforms standard fine-tuning strategies, achieving new state-of-the-art results on child and dialectal speech benchmarks while preserving pre-trained knowledge.
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4\% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
Primary: Argmax, Inc.
All Institutions: Argmax, Inc., University of Virginia, Bilkent University
LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
The paper proposes Localized Attention-Constrained Inference (LACI), an inference-only technique to address reliability issues in long-form autoregressive TTS. The core insight is identifying "emergent alignment heads" within the model's attention mechanism that track the mapping between input text tokens and generated audio tokens. The authors define two specific failure modes: "skips" (deletions) and "hallucinations" (insertions/substitutions). LACI monitors the attention weights of these specific heads in real-time. If the attention pattern deviates from the expected monotonic progress (e.g., skipping a bin of text tokens or stalling), the system detects the error, rolls back the generation state to the error onset, and applies a temporary hard attention mask (guardrail) to force correct alignment during regeneration. This is a clever, low-overhead approach that leverages internal model states rather than external ASR feedback for detection, which is a significant methodological improvement over standard best-of-N sampling.
The experiments are rigorous and well-targeted. The authors use the AppTek Call Center dataset, a real-world long-form TTS use case, which is more relevant than standard short-utterance benchmarks for this specific problem. They evaluate on multiple model scales (Qwen3-TTS-0.6B, 1.7B) and families (VoxCPM2), demonstrating generalizability. The metrics include standard WER and SIM, but the introduction of wSIM (sliding-window speaker similarity) is a valuable contribution that exposes failure patterns hidden by global averaging. The results show dramatic improvements in worst-of-N WER (35.2% to 3.4%) and wSIM (0.01 to 0.47), effectively closing the gap between short-form and long-form reliability. The comparison against ACI (always-on guardrails) clearly demonstrates the superiority of the detection-and-recovery approach.
The paper states that code will be open-sourced upon publication, which is standard for arXiv preprints. However, the specific details of the "instability measure" and detector settings are referenced as being in the code, which is a slight barrier to immediate reproducibility without access to the repository. The use of standard open-weights models (Qwen3-TTS, VoxCPM2) and public datasets (AppTek) aids reproducibility. The manual verification of wSIM failures adds a layer of qualitative validation that is hard to automate but important for trust.
The method relies on the existence of "emergent alignment heads," which may not be present or stable in all TTS architectures, particularly non-autoregressive or diffusion-based models. The paper focuses on autoregressive neural codec language models. The computational overhead, while described as "negligible," involves rollback and regeneration, which could be significant for very long texts if failures are frequent. The evaluation is limited to English (implied by the datasets and models used). The "worst-of-N" metric, while useful for reliability, does not reflect the average user experience if N is large, though the paper argues for the importance of reliability in professional settings.
This work has high practical impact for deploying TTS in real-world applications like audiobooks, customer support agents, and long-form voice cloning. The ability to reliably generate long-form speech with consistent speaker identity is a critical bottleneck for current TTS systems. The proposed wSIM metric is a valuable tool for the community to better evaluate speaker consistency in long-form generation. The inference-only nature of the solution makes it easily adoptable by users of existing open-weights models without retraining. LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.