Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns how, when, and whether the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.
Primary: Alibaba Group
All Institutions: Alibaba Token Foundry, Alibaba Group
The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
The paper proposes a comprehensive framework for real-time voice agents, structured around three layers: Think (foundation post-training), Act (agentic tool use), and Speak/Coordinate (conversational policy). The "Think" layer introduces a sophisticated post-training pipeline combining Core-Cocktail SFT with a novel "M$^2$-OPD" (Multimodality and Multi-Teacher On-Policy Distillation) strategy. This involves using a Text Teacher and a frozen Audio Reference to supervise student-generated trajectories, effectively transferring text-based reasoning capabilities to native audio models while preserving audio-specific nuances. The "Act" layer is particularly strong, introducing self-evolving executable environments for Group Relative Policy Optimization (GRPO). By using code agents to build, validate, and evolve tasks based on model performance (difficulty gating), the system learns robust tool use, state verification, and grounded progress communication. The "Speak/Coordinate" layer formalizes the decision-making process for when to speak, act, or remain silent, addressing critical issues in full-duplex interaction like background speech handling and turn-taking.
The evaluation is extensive, covering intelligence (audio reasoning, multilingual ASR, long-context), action (tool use, retrieval), interaction (persona, empathy, full-duplex behavior), and safety. Key results include a significant improvement in task success on the $\tau$-Voice benchmark (78.4% to 82.0%) and a dramatic reduction in response rate to background speech on Full-Duplex-Bench (73.0% to 13.0%), indicating much better conversational etiquette. The model also shows strong gains in multilingual audio understanding (BBA) and safety metrics. The inclusion of a "Voice Harness" prototype for persistent tasks adds depth to the system-level contribution.
The paper provides detailed descriptions of the training pipeline, data scales (approx. 1M hours), and evaluation protocols. However, as a technical report from a major industry lab, specific hyperparameters, exact dataset compositions, and code are not fully released, limiting independent reproduction. The reliance on in-house benchmarks (Long-AMC, WebSearch1K, VoiceChat) also restricts external verification.
The primary limitation is the lack of open-source code and data, making it difficult for the community to replicate the results. The evaluation relies heavily on in-house benchmarks and automatic judges (e.g., GPT-4o-mini, Qwen-Plus), which may introduce bias. The "Voice Harness" is presented as a prototype and is not fully integrated into the core model evaluation. Additionally, the paper focuses on a specific adaptation of $\tau$-Voice (half-duplex S2T) rather than the official full-duplex S2S protocol, which may limit direct comparability with other full-duplex systems.
This work significantly advances the state-of-the-art in real-time voice assistants by integrating robust agentic capabilities with natural conversational policies. The M$^2$-OPD technique and self-evolving environment for GRPO are valuable contributions to the broader field of multimodal LLM post-training. The focus on safety and reliable interaction (e.g., handling background speech, refusing unsupported requests) is crucial for the deployment of such systems in real-world scenarios. The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Primary: Alibaba Group
All Institutions: The Chinese University of Hong Kong, Alibaba Group, Shanghai Jiao Tong University, Shanghai Innovation Institute, Zhejiang University
The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
The paper introduces a coherent framework for "native" audio-visual dialogue, addressing the critical gap in data availability for models that process raw audio/video without ASR intermediaries. The core methodological contribution is OmniVChat-Studio, a multi-agent synthesis engine. The design is robust, utilizing distinct agents (Director, Renderer, Reviewer, Validator) to handle the complex pipeline of script generation, video rendering, and quality verification. The separation of "Fixed" and "Flexible" modules allows for extensibility to new subcategories without re-engineering the core pipeline. The proposed OmniVChat-RL reward design is particularly strong, moving beyond simple correctness to incorporate efficiency (conciseness) and style (naturalness), which are crucial for conversational agents. The use of GSPO (Group Sequence Policy Optimization) with LoRA is a practical and effective training strategy for large multimodal models.
The experimental setup is rigorous, featuring a synthetic benchmark (OmniVChat-Bench) and a human-recorded transfer probe (OmniVChat-Bench-Human). The evaluation metrics are well-defined, using LLM-based grading with tiered rubrics to handle the open-ended nature of dialogue. The results show significant improvements in both correctness and efficiency (reply length reduction) when using the proposed RL method. The comparison against state-of-the-art models (Gemini, Qwen, etc.) provides a strong baseline. The ablation studies effectively isolate the contributions of the efficiency and style rewards. The transfer from synthetic to human data is a key validation point, and the results support the claim that synthetic data can effectively train models for real-world scenarios.
The paper provides detailed descriptions of the agent roles, sampling strategies, and reward functions. The code and data are promised to be released, which is standard for such contributions. The specific prompts for the Director and Reviewer agents are referenced in appendices, which aids reproducibility. However, the reliance on specific proprietary video generation models (implied by the "Renderer" agent) and LLMs for grading may limit full reproducibility for external groups without access to these specific tools.
The primary limitation is the reliance on synthetic data for training and evaluation. While the paper argues for the validity of this approach, the "reality gap" between synthesized and real-world audio-visual noise, lighting, and human behavior remains a concern, even with the human-recorded probe. The evaluation relies heavily on LLM-as-a-judge, which can introduce biases or inconsistencies, although the paper attempts to mitigate this with detailed rubrics and agreement checks. The multi-turn evaluation is limited to single-turn probes in the human dataset, potentially underestimating the model's performance in long, complex real-world conversations.
This work has significant implications for the development of real-time, low-latency multimodal assistants. By demonstrating that synthetic data can effectively train models for native audio-visual dialogue, it lowers the barrier to entry for developing such systems, which are currently hindered by the scarcity of high-quality, annotated real-world data. The focus on efficiency and style in the reward model aligns with user expectations for natural, concise interactions, making the research highly relevant to industry applications in smart devices and virtual assistants. The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
Primary: University of Maryland
All Institutions: Adobe Research, University of Maryland, OpenAI
[One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
The paper proposes a structured taxonomy of 22 paralinguistic and acoustic characteristics, which is a significant conceptual contribution to the field. The methodology relies heavily on a data simulation engine that augments clean speech with Room Impulse Responses (RIRs) and noise to create controlled acoustic environments. This is a standard but effective approach for acoustic tasks. The core innovation lies in the two-stage curriculum learning strategy: Stage 1 focuses on atomic single-attribute QA pairs to build foundational knowledge, while Stage 2 uses LLM-generated multi-attribute QA pairs to foster joint reasoning. The use of Qwen2.5-7B for generating training data and Mistral-Small-3.2 for generating benchmark questions to avoid self-referential bias is a thoughtful methodological choice. However, the reliance on synthetic data for acoustic characteristics and existing datasets for speaker traits limits the generalizability to real-world, uncontrolled acoustic scenes. The model is fine-tuned from Qwen2-Audio-7B-Instruct using LoRA, which is a standard and efficient approach.
The experimental setup is robust, introducing ParA-Bench, a 6,000-question multiple-choice benchmark. The evaluation compares ParA-LLM against state-of-the-art models like GPT-4o-Audio, Qwen2-Audio, and specialized reasoning models. The results show a 7.5% improvement over GPT-4o-Audio on the proposed benchmark, which is a strong result. The ablation study on the curriculum learning shows consistent gains, validating the training strategy. The inclusion of downstream applications, such as Text-to-Impulse Response (Text2IR) generation, demonstrates the practical utility of the learned representations. However, the evaluation is primarily limited to the proposed benchmark and a few standard audio benchmarks (MMAU-Pro, MMAR). There is a lack of evaluation on other established paralinguistic benchmarks (e.g., IEMOCAP for emotion, or specific speaker verification datasets) to fully validate the model's capabilities across different paralinguistic tasks.
The paper provides detailed information on the data sources (EARS, Emilia, Expresso, VoxCeleb, MIT IR Survey, etc.), the simulation parameters, and the training hyperparameters (LoRA rank, learning rates, batch sizes). The release of the dataset, model, and benchmark significantly enhances reproducibility. The use of specific models for data generation (Qwen2.5-7B, Mistral-Small-3.2) is clearly stated. However, the exact prompts used for LLM-based QA generation are not fully detailed in the text, which could be a minor barrier to exact reproduction of the data pipeline.
The primary limitation is the reliance on synthetic acoustic data for the acoustic characteristics, which may not fully capture the complexity and variability of real-world acoustic environments. The speaker and speech characteristics are derived from existing datasets, which may have inherent biases or limited diversity. The model is based on a 7B parameter LLM, which may limit its scalability to more complex reasoning tasks compared to larger models. The benchmark, while novel, is relatively small (6K questions) and may not fully represent the breadth of paralinguistic understanding. The paper does not extensively discuss the computational cost of the data generation pipeline or the potential biases introduced by the LLMs used for data augmentation.
This work has significant potential impact on the field of audio AI by addressing a critical gap in paralinguistic understanding. The release of a large-scale dataset and a comprehensive benchmark will facilitate future research in this area. The demonstrated downstream applications, such as TTS control and acoustic space captioning, highlight the practical value of the model. The framework could be extended to other modalities or languages, and the curriculum learning approach could be applied to other multi-attribute understanding tasks. The work also contributes to the broader goal of building more human-like audio AI systems that can understand not just what is said, but how it is said. [One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
Sparse AutoEncoders (SAEs) offer a promising way to inspect language model representations, but it is still unclear what kind of linguistic structure their latents expose. We use part-of-speech (PoS) categories as a controlled test case to study whether morpho-syntactic information is encoded by individual latents or by structured groups of features. We find that PoS distinctions are highly recoverable from SAE activations, but do not align with one-to-one latent / category mappings. This recoverability is not reducible to lexical memorisation, and Open and Closed PoS classes differ substantially. Categories are supported by compact groups of sparse latents, with substantial variation across tags. These groups remain stable on held-out data, while also showing overlap between related categories. Our results show that SAEs localise morpho-syntactic information in a distributed and category-dependent form rather than through atomic grammatical features.
Primary: University of Pisa
All Institutions: University of Pisa
The paper demonstrates that Part-of-Speech categories in LLMs are encoded as distributed but compact groups of SAE latents, with significant structural differences between open and closed classes. By combining probing, feature salience, and coverage analysis, the authors provide a detailed map of how morpho-syntactic information is organized in sparse latent spaces, offering valuable insights into the interpretability of modern language models.
The paper employs a rigorous three-step interpretability pipeline to analyze Part-of-Speech (PoS) encoding in Sparse Autoencoder (SAE) latents. First, it establishes recoverability using L1-regularized logistic regression probes on SAE activations from LLaMA-3-8B. Second, it moves beyond binary accuracy to localize information by analyzing feature salience (classifier coefficients) and coverage (the minimal number of latents required to cover 95% of instances for a given PoS tag). Third, it validates these localized groups on held-out data and a controlled synthetic dataset to test stability and additivity. The methodology is sound, moving logically from "is the information there?" to "where is it?" and "is it stable?". The use of a controlled dataset with minimal templates is a strong methodological choice to isolate lexical effects from syntactic ones.
The experiments are comprehensive, utilizing the GUM Treebank for natural data and a custom controlled dataset for validation. The results clearly demonstrate that PoS information is distributed across compact groups of latents rather than single monosemantic units. The distinction between Open-class (nouns, verbs) and Closed-class (determiners, conjunctions) PoS is well-supported, with closed classes showing more compact and stable latent groups. The ablation studies, including random label controls and layer-wise analysis, effectively rule out trivial explanations like lexical memorization or layer-specific artifacts. The finding that a small union of these latents (498 out of 131,072) supports strong multi-class classification is a significant quantitative result.
The authors provide high reproducibility by releasing code, data, and the specific SAE checkpoint used. The GitHub repository contains the pipeline for extracting activations and running the probes. The use of standard libraries (Scikit-learn, Sparsify) and public models (LLaMA-3-8B, EleutherAI SAE) ensures that other researchers can easily replicate the findings. The detailed description of the subword-to-token alignment strategy (leftmost subword anchoring) is crucial for reproducibility in this domain.
The study is limited to a single model (LLaMA-3-8B) and a single SAE variant, which may limit the generalizability of the findings to other architectures or sparsity regimes. The analysis is restricted to English, and it is unclear how these patterns would manifest in morphologically rich languages. The controlled dataset, while useful, is small and covers only a subset of syntactic structures. Additionally, the reliance on linear probes means that non-linear interactions between latents are not captured, potentially underestimating the complexity of the representation.
This work contributes to the broader field of mechanistic interpretability by providing a structured framework for analyzing how linguistic categories are represented in sparse latent spaces. It challenges the assumption of strict monosemanticity for linguistic features, suggesting instead a distributed but localized organization. This has implications for how interpretability tools are designed and how we understand the internal representations of LLMs. The findings could inform the development of more targeted interpretability methods that focus on groups of features rather than individual units. The paper demonstrates that Part-of-Speech categories in LLMs are encoded as distributed but compact groups of SAE latents, with significant structural differences between open and closed classes. By combining probing, feature salience, and coverage analysis, the authors provide a detailed map of how morpho-syntactic information is organized in sparse latent spaces, offering valuable insights into the interpretability of modern language models.
Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns how, when, and whether the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.
Primary: Alibaba Group
All Institutions: Alibaba Token Foundry, Alibaba Group
The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
The paper proposes a comprehensive framework for real-time voice agents, structured around three layers: Think (foundation post-training), Act (agentic tool use), and Speak/Coordinate (conversational policy). The "Think" layer introduces a sophisticated post-training pipeline combining Core-Cocktail SFT with a novel "M$^2$-OPD" (Multimodality and Multi-Teacher On-Policy Distillation) strategy. This involves using a Text Teacher and a frozen Audio Reference to supervise student-generated trajectories, effectively transferring text-based reasoning capabilities to native audio models while preserving audio-specific nuances. The "Act" layer is particularly strong, introducing self-evolving executable environments for Group Relative Policy Optimization (GRPO). By using code agents to build, validate, and evolve tasks based on model performance (difficulty gating), the system learns robust tool use, state verification, and grounded progress communication. The "Speak/Coordinate" layer formalizes the decision-making process for when to speak, act, or remain silent, addressing critical issues in full-duplex interaction like background speech handling and turn-taking.
The evaluation is extensive, covering intelligence (audio reasoning, multilingual ASR, long-context), action (tool use, retrieval), interaction (persona, empathy, full-duplex behavior), and safety. Key results include a significant improvement in task success on the $\tau$-Voice benchmark (78.4% to 82.0%) and a dramatic reduction in response rate to background speech on Full-Duplex-Bench (73.0% to 13.0%), indicating much better conversational etiquette. The model also shows strong gains in multilingual audio understanding (BBA) and safety metrics. The inclusion of a "Voice Harness" prototype for persistent tasks adds depth to the system-level contribution.
The paper provides detailed descriptions of the training pipeline, data scales (approx. 1M hours), and evaluation protocols. However, as a technical report from a major industry lab, specific hyperparameters, exact dataset compositions, and code are not fully released, limiting independent reproduction. The reliance on in-house benchmarks (Long-AMC, WebSearch1K, VoiceChat) also restricts external verification.
The primary limitation is the lack of open-source code and data, making it difficult for the community to replicate the results. The evaluation relies heavily on in-house benchmarks and automatic judges (e.g., GPT-4o-mini, Qwen-Plus), which may introduce bias. The "Voice Harness" is presented as a prototype and is not fully integrated into the core model evaluation. Additionally, the paper focuses on a specific adaptation of $\tau$-Voice (half-duplex S2T) rather than the official full-duplex S2S protocol, which may limit direct comparability with other full-duplex systems.
This work significantly advances the state-of-the-art in real-time voice assistants by integrating robust agentic capabilities with natural conversational policies. The M$^2$-OPD technique and self-evolving environment for GRPO are valuable contributions to the broader field of multimodal LLM post-training. The focus on safety and reliable interaction (e.g., handling background speech, refusing unsupported requests) is crucial for the deployment of such systems in real-world scenarios. The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate "thinking" tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15\% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.
Primary: National Taiwan University
All Institutions: National Taiwan University, NVIDIA Research
The paper introduces a novel RL-based self-refinement framework for speech synthesis that enables a LALM to reason over its own audio output. By combining a generate-critique-refine pipeline with a specifically designed GRPO reward function that accounts for iterative improvement, the method achieves significant gains in instruction-following capability, demonstrating the viability of audio-token reasoning for enhancing generative quality.
The paper proposes a "Listen, Critique, and Refine" framework for instruction-following speech synthesis using Large Audio Language Models (LALMs). The core methodological contribution is extending the chain-of-thought/reasoning paradigm from text to audio tokens. The model generates a draft speech ($v_1$), generates a textual critique ($c$) by "listening" to $v_1$, and then generates a refined speech ($v_2$) conditioned on both. The training utilizes Group Relative Policy Optimization (GRPO) with a novel reward function. A key technical insight is the use of a non-linear transformation for the improvement term in the reward to prevent cancellation of the constant first-pass baseline within the GRPO group-relative advantage calculation. This is a sound and well-motivated technical detail that addresses a specific failure mode in applying GRPO to iterative refinement tasks.
The experiments are conducted on the InstructTTSEval benchmark, covering in-domain (DSD) and out-of-domain (APS, RP) tasks. The evaluation uses both objective metrics (CLSP score, WER) and subjective metrics (LALM-as-a-judge, human evaluation). The results show consistent improvements over zero-shot and RL one-hop baselines. The ablation studies effectively isolate the contributions of the non-linear reward shaping, speaker retrieval, and the critique mechanism. The human evaluation with 5 senior researchers adds credibility to the subjective gains. However, the training dataset is small (1000 examples from ParaSpeechCaps), which raises questions about the generalizability and robustness of the learned refinement behavior.
The authors provide a GitHub repository link, which is a positive step. The paper details the model architecture (Step-Audio-2-mini), hyperparameters, and implementation details (LoRA, vLLM, NCCL). However, the reliance on specific proprietary or less common models (Step-Audio-2-mini, CLSP) might limit reproducibility for the broader community compared to using more standard open-source components. The small training set size is a specific detail that is reproducible but may not be representative of large-scale applications.
The primary limitation is the small training dataset (1000 samples), which may lead to overfitting or limited generalization. The method relies on the base model's ability to generate meaningful critiques, which may not always be accurate or useful. The two-hop inference process doubles the computational cost compared to single-pass generation. The evaluation is limited to English speech. The reliance on CLSP for style adherence may not capture all nuances of complex instructions.
This work demonstrates the potential of self-refinement and reasoning in audio generation, a promising direction for improving the controllability and quality of LALMs. The technique of using non-linear reward shaping to handle iterative refinement under GRPO could be applicable to other multi-step generation tasks in other modalities. The framework provides a blueprint for integrating understanding and generation in a unified reasoning loop. The paper introduces a novel RL-based self-refinement framework for speech synthesis that enables a LALM to reason over its own audio output. By combining a generate-critique-refine pipeline with a specifically designed GRPO reward function that accounts for iterative improvement, the method achieves significant gains in instruction-following capability, demonstrating the viability of audio-token reasoning for enhancing generative quality.
Large audio-language models (LALMs) are increasingly used for a broader range of audio reasoning tasks. These models typically incorporate audio representations into a large language model (LLM) backbone to enable multimodal reasoning. Recent test-time reinforcement learning (TTRL) methods further improve LLM reasoning capability by leveraging unlabelled test data after pre-training. However, the importance of the perceptual capability of LALMs remains underexplored, particularly how much acoustic evidence is integrated and relied upon during reasoning, and how this contributes to final task performance. This gap limits the development of effective post-training methods like TTRL for audio reasoning. In this work, we first analyse how audio information is integrated and utilised during reasoning process. We quantify layer-wise perceptual reliance and show that stronger acoustic reliance is associated with higher accuracy and a larger performance gain attributable to the audio input. Building on this, we propose Perception-Grounded TTRL (PG-TTRL), which aligns label-free test-time optimisation with perceptually grounded reasoning, encouraging the model to structure its reasoning more strongly on the audio input. Experiments across LALMs and benchmarks show that PG-TTRL consistently improves reasoning performance over both the base models and standard TTRL, showing the value of perceptual-grounding optimisation for test-time audio reasoning.
Primary: University of Maryland, College Park
All Institutions: University of Maryland, College Park, University of Illinois Urbana-Champaign, University of Washington
The paper presents a rigorous analysis of perceptual reliance in LALMs and proposes PG-TTRL, a novel test-time reinforcement learning method that aligns policy optimization with acoustic grounding, resulting in consistent performance gains over standard TTRL and base models.
The paper introduces a novel diagnostic framework for Large Audio-Language Models (LALMs) that quantifies "perceptual reliance" by comparing hidden representations under full audio access versus masked audio attention. This analysis reveals that acoustic evidence integration peaks in intermediate layers and correlates with task accuracy. Building on this, the authors propose PG-TTRL (Perception-Grounded Test-Time Reinforcement Learning), which modifies the Group Relative Policy Optimization (GRPO) advantage function. Instead of relying solely on majority vote correctness, PG-TTRL computes a trajectory-level grounding score based on the log-probability gap between original and masked audio conditions. This score is used to derive a reliability weight that scales the advantage, effectively penalizing updates that rely on language priors rather than acoustic evidence. The method is theoretically sound, addressing a critical flaw in standard TTRL where models may reinforce linguistic shortcuts. The integration of causal intervention (attention masking) into the reward signal is a creative and rigorous approach to grounding multimodal reasoning.
Experiments are conducted on two state-of-the-art LALMs (Qwen2.5-Omni-3B and 7B) across two major benchmarks (MMAR and MMAU). The results show consistent improvements over both base models and standard TTRL, with gains up to 4.9 points on MMAU. The analysis of Pass@k metrics is particularly compelling, demonstrating that PG-TTRL preserves diverse correct trajectories better than standard TTRL, which tends to collapse to incorrect majority votes on hard instances. The ablation study on layer-wise changes confirms that the method specifically enhances late-stage reasoning utilization of audio, validating the hypothesis. However, the evaluation is limited to multiple-choice tasks, and the sample size for correlation analysis is modest.
The paper provides detailed hyperparameters, including LoRA ranks, learning rates, and specific bounds for the reliability weight. The algorithm is clearly defined, and the reproducibility checklist is filled out, indicating that code will be released. The use of standard benchmarks and open-source models (Qwen) enhances reproducibility. However, the specific implementation of the attention masking for the grounding score requires careful handling of the model's internal attention mechanisms, which may be non-trivial to replicate without the provided code.
The primary limitation is the restriction to multiple-choice audio reasoning tasks; the method's applicability to open-ended generation or other modalities is not explored. The grounding score relies on teacher-forcing passes, which may not perfectly reflect the autoregressive generation process. Additionally, the method adds computational overhead due to the extra forward passes required for grounding score calculation. The correlation analysis excludes small-sample tasks, which may limit the generalizability of the findings.
This work addresses a fundamental issue in multimodal AI: the tendency of models to ignore perceptual inputs in favor of language priors. By providing a mechanism to explicitly reward perceptual grounding, the paper offers a pathway to more robust and trustworthy multimodal systems. The insights into layer-wise perceptual reliance are valuable for the broader community developing multimodal architectures. The method could be extended to other modalities (vision, video) and tasks, potentially improving the reliability of AI systems in safety-critical applications where accurate perception is crucial. The paper presents a rigorous analysis of perceptual reliance in LALMs and proposes PG-TTRL, a novel test-time reinforcement learning method that aligns policy optimization with acoustic grounding, resulting in consistent performance gains over standard TTRL and base models.
Constraint-following music generation asks a score to satisfy several user-specified properties at once, each checkable programmatically (key, meter, length, range, final note, rhythm, motion and form), yet no existing benchmark isolates this capability. We construct MusicConstraintBench, 2,180 items over eight constraint families, on which current models fail once a few constraints are combined. The natural remedy is reinforcement learning with these verifiers as reward, yet we observe that a reward paid only when every property holds leaves most training groups without a learning signal: over the first 50 updates, 0.550 of rollout groups score identically and receive no gradient, even though a failing score typically misses only one requested property. Under the joint criterion, rollouts for a prompt tend to fail together, so a binary reward cannot separate a nearly correct score from a malformed one. We therefore introduce MusicRLVR, which pays graded per-property credit behind a hard validation gate that rejects malformed outputs, plus a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain fine-tuning. On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints and leads every zero-shot baseline including Llama-3.1-70B at 0.380. It also generalises to property combinations unseen in training and to out-of-range parameter values, showing that verifiable rewards need not presuppose a target output.
Primary: Anonymous (Double-Blind Review)
All Institutions: Anonymous
The paper introduces a verifier-driven reinforcement learning framework and benchmark for constraint-following symbolic music generation, demonstrating that graded, deterministic rewards can significantly improve multi-constraint compliance in large language models without requiring music-domain supervised fine-tuning.
The paper proposes MusicRLVR, a reinforcement learning framework that uses deterministic, programmatic verifiers as rewards for symbolic music generation. The core methodological contribution is the design of a graded reward function that combines a hard validity gate (rejecting malformed ABC notation) with per-family partial credit and a joint-satisfaction bonus. This addresses the "sparse reward" problem in GRPO where binary success/failure signals lead to zero-variance groups and no gradient updates. The approach is technically sound, leveraging the fact that musical constraints (key, meter, form, etc.) are programmatically checkable. The use of GRPO (Group Relative Policy Optimization) is appropriate for this setting, and the ablation studies effectively demonstrate the necessity of the graded credit component over binary rewards.
The authors introduce MusicConstraintBench, a new benchmark of 2,180 items covering eight constraint families. The experiments are rigorous, comparing the proposed method against strong zero-shot baselines (including Llama-3.1-70B) and supervised fine-tuning (SFT) baselines. The results show significant improvements in constraint compliance, particularly on multi-constraint tasks where baselines fail. The evaluation includes tests for compositional generalization (unseen constraint combinations) and parameter extrapolation, which are critical for assessing the robustness of the learned policy. The statistical significance is supported by McNemar tests and Holm correction.
The paper provides extensive details on the training setup, hyperparameters, and data construction. It explicitly states that code, benchmark suites, and model outputs will be released under an MIT license. The reproducibility statement is detailed, mentioning specific scripts for asset generation and evaluation. The use of standard tools like vLLM and ms-swift enhances reproducibility.
The primary limitation is the domain specificity; the method is tailored to symbolic music in ABC notation and may not directly transfer to other domains without significant adaptation. Additionally, the paper acknowledges that compliance scores do not measure perceptual musical quality, which is a significant gap for practical applications. The reliance on a specific corpus (IrishMAN) for SFT data might introduce stylistic biases, although the RL phase is designed to be independent of this.
This work demonstrates that RLVR (Reinforcement Learning with Verifiable Rewards) can be applied to open-ended generation tasks where the output space is large but the properties of interest are verifiable. This has implications for other structured generation tasks (e.g., code, math, structured data) where deterministic checks are possible. It provides a template for building benchmarks and reward functions for property-constrained generation. The paper introduces a verifier-driven reinforcement learning framework and benchmark for constraint-following symbolic music generation, demonstrating that graded, deterministic rewards can significantly improve multi-constraint compliance in large language models without requiring music-domain supervised fine-tuning.
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
Primary: University of Maryland
All Institutions: Adobe Research, University of Maryland, OpenAI
[One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
The paper proposes a structured taxonomy of 22 paralinguistic and acoustic characteristics, which is a significant conceptual contribution to the field. The methodology relies heavily on a data simulation engine that augments clean speech with Room Impulse Responses (RIRs) and noise to create controlled acoustic environments. This is a standard but effective approach for acoustic tasks. The core innovation lies in the two-stage curriculum learning strategy: Stage 1 focuses on atomic single-attribute QA pairs to build foundational knowledge, while Stage 2 uses LLM-generated multi-attribute QA pairs to foster joint reasoning. The use of Qwen2.5-7B for generating training data and Mistral-Small-3.2 for generating benchmark questions to avoid self-referential bias is a thoughtful methodological choice. However, the reliance on synthetic data for acoustic characteristics and existing datasets for speaker traits limits the generalizability to real-world, uncontrolled acoustic scenes. The model is fine-tuned from Qwen2-Audio-7B-Instruct using LoRA, which is a standard and efficient approach.
The experimental setup is robust, introducing ParA-Bench, a 6,000-question multiple-choice benchmark. The evaluation compares ParA-LLM against state-of-the-art models like GPT-4o-Audio, Qwen2-Audio, and specialized reasoning models. The results show a 7.5% improvement over GPT-4o-Audio on the proposed benchmark, which is a strong result. The ablation study on the curriculum learning shows consistent gains, validating the training strategy. The inclusion of downstream applications, such as Text-to-Impulse Response (Text2IR) generation, demonstrates the practical utility of the learned representations. However, the evaluation is primarily limited to the proposed benchmark and a few standard audio benchmarks (MMAU-Pro, MMAR). There is a lack of evaluation on other established paralinguistic benchmarks (e.g., IEMOCAP for emotion, or specific speaker verification datasets) to fully validate the model's capabilities across different paralinguistic tasks.
The paper provides detailed information on the data sources (EARS, Emilia, Expresso, VoxCeleb, MIT IR Survey, etc.), the simulation parameters, and the training hyperparameters (LoRA rank, learning rates, batch sizes). The release of the dataset, model, and benchmark significantly enhances reproducibility. The use of specific models for data generation (Qwen2.5-7B, Mistral-Small-3.2) is clearly stated. However, the exact prompts used for LLM-based QA generation are not fully detailed in the text, which could be a minor barrier to exact reproduction of the data pipeline.
The primary limitation is the reliance on synthetic acoustic data for the acoustic characteristics, which may not fully capture the complexity and variability of real-world acoustic environments. The speaker and speech characteristics are derived from existing datasets, which may have inherent biases or limited diversity. The model is based on a 7B parameter LLM, which may limit its scalability to more complex reasoning tasks compared to larger models. The benchmark, while novel, is relatively small (6K questions) and may not fully represent the breadth of paralinguistic understanding. The paper does not extensively discuss the computational cost of the data generation pipeline or the potential biases introduced by the LLMs used for data augmentation.
This work has significant potential impact on the field of audio AI by addressing a critical gap in paralinguistic understanding. The release of a large-scale dataset and a comprehensive benchmark will facilitate future research in this area. The demonstrated downstream applications, such as TTS control and acoustic space captioning, highlight the practical value of the model. The framework could be extended to other modalities or languages, and the curriculum learning approach could be applied to other multi-attribute understanding tasks. The work also contributes to the broader goal of building more human-like audio AI systems that can understand not just what is said, but how it is said. [One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Existing benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evaluate continuous and discrete representations across speech, sound and music. Utilizing our UniARC framework with dual evaluation strategies across model scales from SmolLM2-135M to Llama-3-8B, we analyze the dynamic relationships of data volume, model capacity, and computational efficiency. Our results reveal the pivotal role of semantic constraints in tokenization for audio understanding and demonstrate that scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks. These findings offer practical guidance for balancing semantic density, fidelity, and efficiency in future LALMs.
Primary: Tsinghua University
All Institutions: Tsinghua University
The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
The paper proposes the UniARC framework, a systematic evaluation methodology for comparing continuous (SSL-based) and discrete (clustering/codec-based) audio representations within Large Audio Language Models (LALMs). The methodology is sound, utilizing two distinct evaluation strategies: parameter-efficient fine-tuning (LoRA) on smaller backbones (SmolLM2) and frozen-backbone probing on larger backbones (Llama-3). This dual approach effectively isolates the impact of representation quality from the capacity of the language model. The choice of encoders is comprehensive, covering major SSL models (HuBERT, WavLM, Whisper) and discrete tokenizers (K-means, DAC, WavTokenizer, SpeechTokenizer). The standardization of feature processing (projectors, temporal concatenation) ensures fair comparison.
The experiments span three domains (speech, sound, music) and multiple tasks (ASR, emotion, captioning, classification). The results provide clear insights: semantic alignment is more critical than raw fidelity for understanding tasks. The finding that scaling the LLM backbone does not compensate for poor audio representation quality is a significant and counter-intuitive result that adds value. The analysis of data diversity and computational efficiency further strengthens the contribution. However, the sample size of encoders is limited, and the specific hyperparameters for the K-means clustering (1000 centroids) might not be optimal for all scales, though the authors justify this choice.
The paper provides sufficient details for reproduction, including model names, dataset names, and general training strategies (LoRA rank, learning rates). However, specific details on the projector architecture (beyond "two-layer MLP") and the exact prompt templates used for instruction tuning are not fully detailed in the provided text. The reliance on open-source models (SmolLM2, Llama-3, XARES-LLM base) aids reproducibility.
The study is limited to specific model scales (up to 8B) and a specific set of encoders. It does not explore hybrid approaches in depth, only comparing pure continuous vs. pure discrete. The "inverse scaling" phenomenon observed might be confounded by the simple MLP projector, as the authors acknowledge. The lack of ablation studies on the projector design is a minor weakness.
The findings have practical implications for the design of future LALMs, suggesting that investment in semantically rich audio encoders is more valuable than simply scaling up the LLM backbone for understanding tasks. This guides resource allocation in the field. The emphasis on multi-domain pre-training for encoders is also a valuable takeaway. The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Primary: Alibaba Group
All Institutions: The Chinese University of Hong Kong, Alibaba Group, Shanghai Jiao Tong University, Shanghai Innovation Institute, Zhejiang University
The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
The paper introduces a coherent framework for "native" audio-visual dialogue, addressing the critical gap in data availability for models that process raw audio/video without ASR intermediaries. The core methodological contribution is OmniVChat-Studio, a multi-agent synthesis engine. The design is robust, utilizing distinct agents (Director, Renderer, Reviewer, Validator) to handle the complex pipeline of script generation, video rendering, and quality verification. The separation of "Fixed" and "Flexible" modules allows for extensibility to new subcategories without re-engineering the core pipeline. The proposed OmniVChat-RL reward design is particularly strong, moving beyond simple correctness to incorporate efficiency (conciseness) and style (naturalness), which are crucial for conversational agents. The use of GSPO (Group Sequence Policy Optimization) with LoRA is a practical and effective training strategy for large multimodal models.
The experimental setup is rigorous, featuring a synthetic benchmark (OmniVChat-Bench) and a human-recorded transfer probe (OmniVChat-Bench-Human). The evaluation metrics are well-defined, using LLM-based grading with tiered rubrics to handle the open-ended nature of dialogue. The results show significant improvements in both correctness and efficiency (reply length reduction) when using the proposed RL method. The comparison against state-of-the-art models (Gemini, Qwen, etc.) provides a strong baseline. The ablation studies effectively isolate the contributions of the efficiency and style rewards. The transfer from synthetic to human data is a key validation point, and the results support the claim that synthetic data can effectively train models for real-world scenarios.
The paper provides detailed descriptions of the agent roles, sampling strategies, and reward functions. The code and data are promised to be released, which is standard for such contributions. The specific prompts for the Director and Reviewer agents are referenced in appendices, which aids reproducibility. However, the reliance on specific proprietary video generation models (implied by the "Renderer" agent) and LLMs for grading may limit full reproducibility for external groups without access to these specific tools.
The primary limitation is the reliance on synthetic data for training and evaluation. While the paper argues for the validity of this approach, the "reality gap" between synthesized and real-world audio-visual noise, lighting, and human behavior remains a concern, even with the human-recorded probe. The evaluation relies heavily on LLM-as-a-judge, which can introduce biases or inconsistencies, although the paper attempts to mitigate this with detailed rubrics and agreement checks. The multi-turn evaluation is limited to single-turn probes in the human dataset, potentially underestimating the model's performance in long, complex real-world conversations.
This work has significant implications for the development of real-time, low-latency multimodal assistants. By demonstrating that synthetic data can effectively train models for native audio-visual dialogue, it lowers the barrier to entry for developing such systems, which are currently hindered by the scarcity of high-quality, annotated real-world data. The focus on efficiency and style in the reward model aligns with user expectations for natural, concise interactions, making the research highly relevant to industry applications in smart devices and virtual assistants. The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: \texttt{
Primary: Meta Reality Labs
All Institutions: Meta Reality Labs
The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
The paper proposes Interrupt and Silent Modeling (ISM), a model-agnostic framework that enables proactive audio assistance in AudioLLMs. The core method involves extending the vocabulary of a standard AudioLLM (Qwen2-Audio-7B) with two special tokens,
The experiments are conducted on ESC-50 (training and evaluation) and Epic-Sounds (zero-shot evaluation). The results on ESC-50 are exceptionally high (99.6% F1), which is somewhat suspiciously perfect for a real-world audio task, suggesting potential overfitting or a very easy evaluation setup (e.g., clean, isolated clips). The zero-shot transfer to Epic-Sounds is a strong point, showing the model generalizes to noisy, egocentric kitchen audio without domain-specific training. The streaming evaluation confirms real-time viability with 3.5s latency. However, the lack of comparison with other proactive or streaming audio models (if any exist) and the limited number of baselines (Zero-Shot, Reactive SFT) weakens the comparative analysis. The "perfect" de-duplication recall is also a red flag that warrants scrutiny of the evaluation protocol.
The paper provides sufficient details on the architecture (Qwen2-Audio-7B, LoRA rank 8), training procedure (two-stage SFT), and data construction for the four states. However, the exact prompts used for the "watch-out intent" and the specific construction of the streaming evaluation set are not fully detailed in the provided text. The code and data are not explicitly linked, which hinders immediate reproducibility.
The primary limitation is the computational cost of using a 7B LLM for a binary decision task, which may be prohibitive for battery-powered wearable devices despite the latency claim. The near-perfect scores on ESC-50 may not reflect real-world performance in highly complex, overlapping acoustic scenes. The paper focuses only on explicit intents, leaving implicit and semantic assistance for future work. The lack of user studies with DHH participants limits the validation of the system's practical utility.
This work has significant potential for assistive technology, particularly for Deaf and Hard of Hearing (DHH) users who rely on sound awareness. By enabling proactive, intent-based alerts with de-duplication, it addresses key pain points in current wearable sound awareness systems (notification fatigue, lack of personalization). The model-agnostic nature of ISM makes it a portable capability that can be integrated into future AudioLLMs, potentially driving a shift from reactive to proactive audio AI. The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
The success of Large Audio Language Models has driven the development of massive multimodal networks exceeding billions of parameters. However, the demand for privacy-preserving, low-latency processing has shifted focus toward Small Audio Language Models (SALMs) capable of on-device execution. In this paper, we introduce Samsone, a family of SALMs designed for edge computing. Our core model, Samsone-134M, establishes a new state-of-the-art for its size class across multiple benchmarks. We further explore the scaling laws of SALMs by introducing Samsone-99M and Samsone-356M. Despite their compact footprint, the Samsone family delivers performance competitive with models orders of magnitude larger. To foster open research and reproducibility, we train Samsone on publicly available data. We release the training code, model weights, mobile-optimized checkpoints and provide an open-source Android application to demonstrate real-time on-device inference of Samsone.
Primary: Samsung R&D Institute Poland
All Institutions: Samsung R&D Institute Poland, AGH University of Kraków
The paper introduces Samsone, a family of small audio language models (SALMs) that achieve state-of-the-art performance for their size class by applying vocabulary reduction and depth pruning to standard encoder-decoder architectures. While the architectural components are not novel, the rigorous engineering of these models for on-device inference, combined with strong benchmark results that challenge the necessity of massive parameter counts for audio understanding, represents a significant practical contribution to the field of efficient multimodal AI.
The methodology relies on a standard "encoder-projector-decoder" architecture, which is not novel in itself. The core technical contribution lies in the systematic application of size optimization techniques—specifically vocabulary reduction and depth pruning—to existing open-source components (Whisper-Tiny encoder and SmolLM2 backbone). The approach is pragmatic rather than architecturally groundbreaking. The use of a SEP token to handle multiple audio inputs is a sensible design choice for a general-purpose audio LM, but it is not a significant theoretical advance. The training procedure is straightforward single-stage fine-tuning, lacking the complex multi-stage pre-training or alignment strategies seen in larger LALMs.
The experimental evaluation is the strongest aspect of the paper. The authors provide a comprehensive comparison against both Large Audio Language Models (LALMs) and existing Small Audio Language Models (SALMs) on rigorous benchmarks like MMAU and MMAU-Pro. The results are compelling: Samsone-134M significantly outperforms previous SALMs (Mellow, Pengi) and approaches the performance of models 60x larger (Qwen2-Audio). The inclusion of on-device latency benchmarks on a specific hardware (Samsung Galaxy S25 Ultra) adds practical value, validating the "on-device" claim. However, the ablation study is somewhat limited, testing only a few alternative components without deep diving into the impact of specific training data mixes or the exact trade-offs of the pruning strategies beyond the final performance numbers.
High. The authors explicitly state they release training code, model weights, and an Android application. The use of publicly available datasets (ReasonAQA, AudioSkillsXL) and open-source base models (Whisper, SmolLM2) ensures that the work can be easily reproduced by the community. The detailed description of the training setup (optimizer, learning rate, epochs) further supports reproducibility.
The paper acknowledges that the models suffer from a loss of general-purpose linguistic capabilities due to extensive AQA fine-tuning. This is a common issue in domain-specific fine-tuning but limits the model's versatility. Additionally, the evaluation is heavily skewed towards multiple-choice and short-answer tasks (AQA); performance on open-ended generation or complex reasoning tasks beyond the benchmark scope is less clear. The lack of hardware-specific optimizations (NPU/GPU) means the reported latency may not represent the best possible performance on modern mobile devices.
This work is highly relevant to the growing field of edge AI and privacy-preserving audio processing. By demonstrating that sub-100M parameter models can achieve competitive performance with billion-parameter models, it lowers the barrier to entry for deploying advanced audio understanding on consumer devices. This has implications for real-time applications in healthcare, smart home assistants, and mobile interfaces where data privacy and latency are critical. The paper introduces Samsone, a family of small audio language models (SALMs) that achieve state-of-the-art performance for their size class by applying vocabulary reduction and depth pruning to standard encoder-decoder architectures. While the architectural components are not novel, the rigorous engineering of these models for on-device inference, combined with strong benchmark results that challenge the necessity of massive parameter counts for audio understanding, represents a significant practical contribution to the field of efficient multimodal AI.