Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
Primary: Universitat Politècnica de València
All Institutions: Universitat Politècnica de València, University of Oxford, Instituto de Telecomunicações
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institute, MOSI Intelligence, Shanghai Jiao Tong University
OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
The paper proposes OmniVAE, a jointly trained audio-video Variational Autoencoder (VAE) designed to address the lack of cross-modal alignment in standard modality-specific tokenizers. The core methodological contribution lies in two training-only objectives added to standard reconstruction losses: (1) Segment-level Audio-Video Contrastive Learning (AVCLIP), which aligns audio and video latents at a fine temporal granularity using a hierarchical negative sampling strategy (intra-clip, sibling-clip, cross-video), and (2) Modality-Specific Semantic Distillation, which injects semantic structure from pretrained Qwen3-Omni encoders into each latent space independently. The approach is technically sound, leveraging established contrastive learning frameworks (InfoNCE) and distillation techniques (iREPA-style) but applying them specifically to the joint audio-video latent space. The three-stage training schedule (pretrain reconstruction -> joint contrastive/distillation -> adversarial refinement) is a practical engineering solution to handle the conflicting optimization dynamics of reconstruction and alignment.
The experimental evaluation is comprehensive and well-structured. The authors conduct extensive ablations to isolate the contributions of reconstruction, distillation, and contrastive learning. They employ a novel "sync probing" task (predicting temporal offset) as a proxy for alignment, which provides strong quantitative evidence that the latent spaces are indeed better aligned than baselines. Downstream text-to-audio-video generation experiments on Verse-Bench show consistent improvements in cross-modal alignment metrics (DeSync, LSE-C) and generation quality (AudioBox, IS, FAD) when using OmniVAE. The use of a large-scale, curated dataset (23M clips) with LLM-generated captions adds significant weight to the empirical claims. The reconstruction quality remains competitive with state-of-the-art independent VAEs, demonstrating that alignment does not come at the cost of fidelity.
The paper provides detailed architectural descriptions, loss formulations, and training schedules. The parameter counts are explicitly listed, distinguishing between inference and training-only modules. The dataset construction pipeline is described, though the specific filtering thresholds and LLM prompts might vary. The reliance on proprietary or large pretrained models (Qwen3-Omni, Wan2.2, HunyuanVideo-Foley) for initialization and distillation is noted, which aids reproducibility for those with access to these resources. The code is not explicitly linked in the text, but the methodology is described with sufficient detail for replication.
The primary limitation is that the alignment is learned during training only; the model does not enforce a shared latent distribution, meaning the latents are still modality-specific but aligned in a shared embedding space via the contrastive head. This might limit direct cross-modal generation without additional mapping layers. The sync probing metric, while useful, is a proxy and may not perfectly correlate with human perception of synchronization in complex scenes. The dataset, while large, is web-scraped and may contain noise despite filtering. The reliance on Qwen3-Omni for distillation assumes the availability of this specific large model, which might not be accessible to all researchers.
OmniVAE contributes to the field of unified multimodal generation by providing a stronger foundational representation for audio-video tasks. By decoupling the alignment learning from the generative model, it allows downstream generators to focus on content creation rather than learning synchronization from scratch. This could accelerate the development of high-quality, synchronized audio-video generative models for entertainment, education, and simulation. The emphasis on fine-grained temporal alignment addresses a key challenge in current generative AI, potentially leading to more realistic and usable synthetic media. OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
Primary: Universitat Politècnica de València
All Institutions: Universitat Politècnica de València, University of Oxford, Instituto de Telecomunicações
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Large audio-language models (LALMs) convey acoustic evidence to language decoders through native audio tokens, yet the internal roles of these tokens remain poorly understood. Using temporal audio grounding as a diagnostic setting, we examine how language-model fine-tuning affects the layerwise semantics, decoder accessibility, and temporal output alignment of native audio-token states through four complementary analyses: query-conditioned token semantics, calibrated token readout, temporal-window probes, and residual-delta erasure during generation. Alongside substantial improvements in temporal localization, semantic analysis of Qwen2.5-Omni shows that latent evidence for queried events is already present before fine-tuning and that the audio tokens most strongly aligned with the queried event appear at similar temporal positions before and after fine-tuning. After fine-tuning, event-related information in audio tokens becomes more accessible to the decoder, especially in early and middle layers, and a cross-checkpoint control shows that this improvement arises primarily from decoder adaptation. Temporal probes show that the base checkpoint already contains recoverable information about annotated windows and that fine-tuning mainly improves alignment with each checkpoint's own predicted temporal support. Residual-delta erasure further shows that removing audio-token updates within predicted windows harms timestamp generation more than removing the same number of randomly selected updates. The same broad improvements in decoder readability and prediction alignment also appear in Qwen2-Audio. Together, these results support a semantics-to-readout account in which grounding fine-tuning helps the decoder read existing event evidence and connect it more reliably to temporal outputs.
Primary: Unknown
All Institutions: Unknown
This paper provides a rigorous mechanistic analysis of how fine-tuning enhances temporal grounding in Large Audio-Language Models, demonstrating that improvements stem primarily from enhanced decoder accessibility to latent event evidence rather than the creation of new semantic representations.
The paper proposes a comprehensive mechanistic interpretability framework for Large Audio-Language Models (LALMs), specifically focusing on the "semantics-to-readout" hypothesis. The methodology is rigorous and multi-faceted, employing four distinct diagnostic tools: (1) Query-Conditioned Token Semantics to measure alignment between audio tokens and text event phrases; (2) Calibrated Token Readout using hidden-state insertion (patching) to assess decoder accessibility; (3) Output-Facing Temporal Diagnostics using linear probes on both annotated and predicted windows; and (4) Residual-Delta Erasure to test the functional necessity of specific token updates during generation. This combination of probing, patching, and causal intervention provides a robust separation of representation quality from decoder utility. The approach is well-suited for the domain, addressing the "black box" nature of multimodal alignment in LALMs.
The experiments are conducted on two prominent LALMs, Qwen2.5-Omni and Qwen2-Audio, fine-tuned on a newly constructed dataset, AudioGrounding-QA, derived from the AudioGrounding corpus. The evaluation demonstrates substantial improvements in temporal grounding metrics (mIoU, F1) post-fine-tuning. The core findings are supported by detailed layer-wise analyses. Key results include the discovery that latent event evidence exists in base models but is not effectively read out, and that fine-tuning primarily enhances decoder accessibility (particularly in early/middle layers) rather than creating new semantic representations. The cross-checkpoint controls and residual erasure results strongly support the authors' claims about the mechanism of improvement. The replication on Qwen2-Audio adds significant weight to the generalizability of the findings.
The paper provides detailed descriptions of the diagnostic pipelines, including the construction of the semantic bank, the specific prompts for readout, and the implementation of linear probes and erasure interventions. The dataset construction is clearly explained, and the fine-tuning setup (LoRA on LM blocks only) is specified. While the full code is not explicitly linked in the text provided, the methodological detail is sufficient for reproduction by experts in the field. The use of standard metrics (mIoU, F1) and established embedding models (SBERT) further aids reproducibility.
The study focuses primarily on temporal grounding, a specific task, which may limit the generalizability of the "semantics-to-readout" account to other multimodal tasks (e.g., audio captioning or open-ended dialogue). The analysis is largely confined to Qwen-based architectures; while Qwen2-Audio is used for replication, the broader landscape of LALMs (e.g., Whisper-based or other proprietary models) is not explored. Additionally, the "calibrated readout" relies on SBERT embeddings, which may not perfectly capture the semantic nuances required for complex event reasoning, potentially underestimating or overestimating readout gains. The residual erasure is limited to layer-level interventions, which might miss more complex, distributed circuits.
This work contributes significantly to the field of mechanistic interpretability for multimodal models. By demystifying how fine-tuning affects internal representations, it provides actionable insights for improving LALM training strategies, particularly for tasks requiring precise temporal or spatial grounding. It challenges the assumption that fine-tuning creates new knowledge from scratch, suggesting instead that it optimizes access to pre-existing latent structures. This has implications for efficient fine-tuning, model compression, and the development of more reliable multimodal AI systems. This paper provides a rigorous mechanistic analysis of how fine-tuning enhances temporal grounding in Large Audio-Language Models, demonstrating that improvements stem primarily from enhanced decoder accessibility to latent event evidence rather than the creation of new semantic representations.
This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.
Primary: Hanyang University
All Institutions: Hanyang University
This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
The paper presents a highly specialized, dependency-free C runtime (`faster-enhancer.c`) designed to deploy the FastEnhancer-Medium speech enhancement model on commodity CPUs. The methodology focuses on extreme engineering optimization rather than algorithmic novelty. Key technical strategies include: (1) Training-free int8 quantization with per-frame activation range recomputation, eliminating the need for calibration datasets; (2) End-to-end runtime specialization with six SIMD tiers (ARM NEON, DOTPROD, I8MM, x86 AVX2, AVX-VNNI, AVX-512 VNNI) selected at initialization; (3) Use of Winograd F(2,3) for $k=3$ convolutions to reduce multiply-accumulate counts; (4) Fusion of GRU gates and dequantization epilogues to minimize memory traffic; (5) Strict byte-identical output guarantees across different compilers and architectures by addressing floating-point non-determinism in the Hann window generation and integer overflow edge cases (clamping to [-127, 127]). The approach is a masterclass in low-level systems engineering for ML deployment.
The evaluation is rigorous and addresses the specific constraints of audio processing. The authors measure Real-Time Factor (RTF) and energy efficiency (joules per second) on two distinct platforms: Apple M2 (macOS) and Galaxy S23+ (Android). They demonstrate a 3.3x speedup over fp32 ONNX Runtime on the M2 (0.069 RTF vs 0.230 RTF). Crucially, they introduce a "duty-cycled" benchmarking protocol that accounts for the 6.67 ms frame deadline, revealing that "race-to-idle" strategies are energy-inefficient compared to pacing. They also evaluate quality using PESQ, STOI, SNR, SI-SDR, and SIGMOS, showing negligible degradation (-0.006 PESQ, -0.08 dB SNR) compared to the fp32 baseline. The analysis of long-term stability (30-minute runs) highlights scheduler-induced jitter, a critical insight for real-world deployment.
The paper provides a high degree of reproducibility. The code is released as a dependency-free library on GitHub. The authors provide detailed descriptions of the quantization scheme, kernel implementations, and the specific steps taken to ensure bit-identical outputs across platforms (e.g., double-precision window generation). The evaluation protocol is clearly defined, including the use of VoiceBank-DEMAND and specific timing methodologies. The inclusion of hash comparisons for output verification adds a layer of rigor often missing in systems papers.
The primary limitation is the scope of the model: it is a port of an existing architecture (FastEnhancer-Medium) and does not propose a new model. The x86 tiers are verified for bit-equality but not timed due to lack of hardware, limiting the generality of the performance claims. The runtime is single-threaded and CPU-only, excluding GPU/TPU acceleration which is common in other deployment stacks. The quality metrics, while robust, rely on standard objective measures; subjective listening tests are not included, though the objective deltas are argued to be inaudible. The "cheapest core" placement on Android was found to miss deadlines, limiting the flexibility of core placement in heterogeneous systems.
This work has significant impact for on-device AI, particularly in privacy-sensitive applications like real-time communication, hearing aids, and voice assistants where latency and power consumption are critical. By providing a lightweight, dependency-free runtime, it lowers the barrier to deploying high-quality neural speech enhancement on resource-constrained devices. It challenges the industry's reliance on heavy general-purpose inference engines (like ONNX Runtime) for specific, fixed models, advocating for specialized, hand-tuned kernels. The emphasis on energy efficiency and deadline adherence provides a valuable framework for evaluating the true cost of ML deployment in audio pipelines. This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
Voice activity detection (VAD) triggers downstream speech processing in always-on systems under strict memory, latency, and compute constraints. Recent compact models report strong accuracy but rely on components that are not widely supported: learnable filterbanks, recurrent layers, or non-causal post-processing. We propose kiloVAD, designed for embedded inference using standard Mel features, CNN-only layers, and tunable context/spectral parameters. We introduce per-layer structured pruning with self-distillation and angle-based quantization-aware training (QAT) that outperforms standard QAT by 1-4%. Evaluated per-frame under causal conditions, kiloVAD achieves 0.850 AUC on AVA-Speech with 2.1 k parameters and 200 ms context, establishing a new state of the art for causal, deployment-ready VAD.
Primary: Analog Devices, Inc.
All Institutions: Analog Devices, Inc., University of California, Los Angeles (UCLA)
The paper presents a well-engineered, deployment-focused VAD model that achieves a strong balance between accuracy, latency, and model size through effective structured pruning and angle-aware quantization.
The paper proposes kiloVAD, a compact Voice Activity Detection (VAD) model designed specifically for edge deployment. The core methodological contributions are architectural and training-related rather than novel deep learning architectures. The architecture is a standard CNN backbone (depthwise separable convolutions) with specific engineering choices: an adapter layer for channel decoupling, global average pooling for context flexibility, and amplitude-agnostic preprocessing. The primary technical novelty lies in the compression pipeline: (1) Per-layer structured pruning optimized via multi-objective search (FPR vs. parameter count) combined with self-distillation, and (2) Angle-based Quantization-Aware Training (QAT). The angle-based QAT modifies the loss function to align feature vectors with class weight vectors (prototypes) and repel them from non-target prototypes, aiming to mitigate quantization-induced angular errors. While the application of angle-based margin losses to QAT is a known technique in classification, its specific adaptation for ultra-low-bitwidth VAD on microcontrollers is a valid engineering contribution. However, the underlying model architecture is not novel; it relies on standard, well-established components (Mel spectrograms, CNNs, Global Average Pooling).
The evaluation is rigorous in the context of edge deployment constraints. The authors evaluate on the AVA-Speech dataset, which is appropriate for testing generalization to unseen environments. They provide a detailed comparison with prior compact VAD models (MarbleNet, TinyVAD, SincQDR, ResectNet, AtomicVAD). A key strength is the strict causal evaluation protocol (200ms context), which highlights the latency advantages of their design over competitors that use non-causal windows or larger contexts. The results show that kiloVAD achieves competitive AUC (0.850) with significantly fewer parameters (2.1k) and lower latency than most competitors, while maintaining full causal compatibility and standard operator support. The ablation studies on pruning strategies and quantization methods are clear and demonstrate the efficacy of the proposed techniques. The claim of "state of the art" is qualified by the specific constraints (causal, standard ops, low latency), which is honest and accurate.
The paper provides sufficient detail for reproduction. The architecture is described with specific layer types, kernel sizes, and channel dimensions. The training protocol includes optimizer settings, learning rate schedules, and loss functions. The authors provide pretrained weights and a web-based demo. The use of standard libraries (torch-pruning, Optuna) and standard datasets (LibriSpeech, AVA-Speech) facilitates reproducibility. The only minor gap is the exact code implementation of the angle-based QAT loss, though the mathematical formulation is provided.
The paper focuses heavily on parameter count and latency, but does not provide detailed energy consumption measurements (e.g., mJ/inference) on specific hardware, which is crucial for "always-on" edge devices. The model's performance on highly non-stationary noise or very low SNR conditions is not deeply explored beyond the standard AVA-Speech evaluation. The reliance on Mel spectrograms, while standard, means it does not explore raw audio end-to-end, which some recent works argue is beneficial for extreme compression. The "angle-based" QAT, while effective, adds complexity to the training loop compared to standard STE QAT.
This work has significant practical impact for the deployment of speech processing systems on resource-constrained IoT devices, smart speakers, and hearables. By providing a model that is compatible with standard embedded toolchains (TFLM) and requires no custom DSP kernels, it lowers the barrier to entry for developers. The emphasis on causal, low-latency inference is critical for user experience in real-time applications. The work contributes to the broader trend of efficient AI, demonstrating that high performance can be achieved through careful engineering and compression techniques rather than just larger models. The paper presents a well-engineered, deployment-focused VAD model that achieves a strong balance between accuracy, latency, and model size through effective structured pruning and angle-aware quantization.
Some text-to-speech systems ship a synthesis model and preset style vectors but not the reference encoder that turns audio into such a vector. The model still accepts a style vector; a user with a voice of their own cannot produce one. We solve for that input directly, inverting the released pipeline by gradient descent: every weight stays frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording. Because the objective discards the time axis, the synthesized text may differ from the recording, so no transcript and no alignment are needed. On 154 speakers from two corpora, ECAPA-TDNN similarity rises from 0.132 to 0.413 and ResNet from 0.099 to 0.401, improving for every speaker; a verifier at its equal-error point accepts 53% of the recovered voices as the target, against 1% for the presets they start from.
Primary: Hanyang University
All Institutions: Hanyang University
This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
The paper proposes a gradient-based inverse optimization method to extract style vectors from frozen Text-to-Speech (TTS) models when the reference encoder is withheld. The core innovation lies in bypassing the need for transcribed reference audio or forced alignment by using a content-independent loss function. Specifically, it optimizes the style vector to match the time-pooled mean and standard deviation of hidden states from an early layer (Layer 4) of a frozen WavLM model. This approach effectively decouples speaker identity from linguistic content, allowing the system to work with untranscribed, arbitrary audio recordings. The method is technically sound, leveraging existing differentiable wrappers for ONNX models and standard optimization techniques (Adam, learning rate scheduling). It draws strong parallels to GAN inversion and Textual Inversion in the image domain, adapting these concepts to the audio TTS space.
The evaluation is conducted on 154 speakers from two corpora (VCTK and Common Voice subset of Seed-TTS Eval). The results demonstrate a significant improvement in speaker similarity metrics (ECAPA-TDNN and ResNet) compared to using the nearest preset style vector. The improvement is universal across all tested speakers. The paper provides a thorough analysis of the stopping criterion, characterizing the "content floor" to prevent overfitting into unrealistic style regions. It also evaluates intelligibility (WER) and naturalness (UTMOS), showing that the extracted voices remain intelligible and natural-sounding. The use of independent verification metrics (ECAPA, ResNet) to validate the optimization objective (WavLM statistics) strengthens the claim of successful voice cloning.
The authors release code and provide detailed hyperparameters, including the optimization setup, loss function definition, and stopping criteria. The use of public models (SupertonicTTS, WavLM, ECAPA-TDNN) and public datasets (VCTK, Common Voice) ensures that the experiments are fully reproducible. The description of the gradient flow through the ONNX modules is sufficiently detailed for implementation.
The method is currently validated on a single TTS model (SupertonicTTS). While the authors argue it should generalize to other models with continuous style inputs, this has not been empirically demonstrated. The optimization is sensitive to the choice of the WavLM layer and the stopping threshold, which are somewhat heuristic. The method does not recover duration/prosody styles, inheriting them from the initialization preset. Furthermore, the evaluation prompts are the same ones used during optimization, so generalization to unseen text is not fully tested. The paper acknowledges that the extracted voices might still be detectable by spoofing countermeasures, even if they pass speaker verification.
This work has significant dual-use implications. On one hand, it highlights a security vulnerability in TTS systems that withhold reference encoders, demonstrating that partial releases do not fully protect voice identity. This can inform better security practices for TTS providers. On the other hand, it provides a powerful tool for voice cloning, which can be misused for impersonation and fraud. The authors address this by releasing code with ethical warnings and not packaging it as a turnkey cloning tool. The work contributes to the broader understanding of latent spaces in generative models and the transferability of self-supervised speech representations. This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
Contrastive Language-Audio Pretraining (CLAP) learns aligned text and audio representations in a shared embedding space. However, independent encoding of each modality limits its ability to model cross-modal semantics in complex audio understanding and retrieval tasks. To address this limitation, this paper proposes Text-Prompted CLAP (TP-CLAP), a parameter-efficient extension of CLAP that introduces a cross-attention-based fusion module to incorporate textual prompts into audio features. TP-CLAP is trained using an audio multiple-choice question (audio-MCQ) framework, where it learns to align query-conditioned audio representations with text embeddings of correct answer choices via contrastive learning. Experiments demonstrate that TP-CLAP achieves competitive performance on audio question answering (audio-QA) with substantially larger audio-LLMs, while also improving the base CLAP model on audio-text retrieval and zero-shot classification benchmarks. The learned representations are further fine-tuned for attribute-focused audio-to-audio retrieval, showing that TP-CLAP consistently outperforms the standard CLAP baseline in music retrieval tasks.
Primary: University of Cambridge
All Institutions: University of Cambridge, Toshiba Cambridge Research Laboratory
This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
The paper proposes TP-CLAP, a parameter-efficient extension of the CLAP model that introduces a cross-attention-based fusion module to condition audio representations on textual prompts. The core methodological contribution lies in the training framework: it leverages AudioMCQ data to train the model to align query-conditioned audio embeddings with the text embeddings of correct multiple-choice answers. This is combined with standard CLAP contrastive loss and auxiliary losses to maintain general alignment. The approach is technically sound, addressing a specific limitation of static embeddings in CLAP (inability to handle query-dependent tasks like QA or attribute-focused retrieval). The use of cross-attention to inject textual context into audio features is a known pattern (similar to Q-Former in LLaVA or cross-attention in BLIP), but its application specifically to enable attribute-focused audio retrieval via an MCQ framework is a distinct and practical contribution. The derivation of the loss functions, including the specific handling of multi-label attributes in the fine-tuning stage, is rigorous.
The experimental evaluation is comprehensive, covering audio-text retrieval, zero-shot classification, audio question answering (QA), and attribute-focused audio-to-audio retrieval. The results show consistent improvements over the base CLAP model and the CLAP+AudioMCQ baseline (which lacks the fusion module). Notably, TP-CLAP achieves competitive performance on audio QA against much larger audio-LLMs, which is a strong result highlighting the efficiency of the approach. The attribute-focused retrieval experiments are particularly compelling, demonstrating that the model can isolate specific attributes (e.g., instrument, genre) rather than relying on global similarity. The use of standard benchmarks (AudioCaps, Clotho, ESC50, FSD50K, NSynth, MagnaTagATune) adds credibility. However, the improvement in standard retrieval tasks is modest, suggesting the primary value is in the conditional/attribute-specific capabilities.
The paper provides detailed implementation details, including encoder architectures (CED-Base, bert-base-uncased), hyperparameters (learning rates, batch sizes, warmup epochs), and loss function weights. The data sources (AudioMCQ, AudioCaps, Clotho, WavCaps, MACS, NSynth, MagnaTagATune) are well-known and accessible. The training stages are clearly delineated. This level of detail allows for reasonable reproducibility, assuming access to the pre-trained CLAP weights and the specified datasets.
The paper acknowledges that the model is smaller than audio-LLMs like Audio Flamingo 3, which still outperform it in QA. The improvements on standard retrieval tasks are marginal. The method relies heavily on the quality of the AudioMCQ dataset; if the questions/answers are noisy or biased, the model may learn spurious correlations. Furthermore, the cross-attention module adds computational overhead during inference compared to the original CLAP, although it remains lightweight compared to full LLMs. The paper does not discuss potential biases in the attribute-focused retrieval (e.g., gender or cultural biases in instrument/genre associations).
This work contributes to the field of multimodal audio understanding by enabling more flexible and query-dependent audio representation learning. It facilitates applications such as advanced music information retrieval (MIR), where users might search for "sad piano music" or "jazz with saxophone," rather than just general genre or mood. It also improves accessibility tools that need to understand specific audio contexts. The parameter-efficient nature makes it more accessible for deployment on edge devices or with limited compute compared to large multimodal LLMs. This paper presents a method for learning query-conditioned audio representations by integrating a cross-attention fusion module into the CLAP framework, trained via an audio multiple-choice question paradigm, demonstrating significant improvements in attribute-focused retrieval and competitive performance in audio question answering tasks.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Primary: Fudan University
All Institutions: Fudan University, Shanghai Innovation Institute, MOSI Intelligence, Shanghai Jiao Tong University
OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
The paper proposes OmniVAE, a jointly trained audio-video Variational Autoencoder (VAE) designed to address the lack of cross-modal alignment in standard modality-specific tokenizers. The core methodological contribution lies in two training-only objectives added to standard reconstruction losses: (1) Segment-level Audio-Video Contrastive Learning (AVCLIP), which aligns audio and video latents at a fine temporal granularity using a hierarchical negative sampling strategy (intra-clip, sibling-clip, cross-video), and (2) Modality-Specific Semantic Distillation, which injects semantic structure from pretrained Qwen3-Omni encoders into each latent space independently. The approach is technically sound, leveraging established contrastive learning frameworks (InfoNCE) and distillation techniques (iREPA-style) but applying them specifically to the joint audio-video latent space. The three-stage training schedule (pretrain reconstruction -> joint contrastive/distillation -> adversarial refinement) is a practical engineering solution to handle the conflicting optimization dynamics of reconstruction and alignment.
The experimental evaluation is comprehensive and well-structured. The authors conduct extensive ablations to isolate the contributions of reconstruction, distillation, and contrastive learning. They employ a novel "sync probing" task (predicting temporal offset) as a proxy for alignment, which provides strong quantitative evidence that the latent spaces are indeed better aligned than baselines. Downstream text-to-audio-video generation experiments on Verse-Bench show consistent improvements in cross-modal alignment metrics (DeSync, LSE-C) and generation quality (AudioBox, IS, FAD) when using OmniVAE. The use of a large-scale, curated dataset (23M clips) with LLM-generated captions adds significant weight to the empirical claims. The reconstruction quality remains competitive with state-of-the-art independent VAEs, demonstrating that alignment does not come at the cost of fidelity.
The paper provides detailed architectural descriptions, loss formulations, and training schedules. The parameter counts are explicitly listed, distinguishing between inference and training-only modules. The dataset construction pipeline is described, though the specific filtering thresholds and LLM prompts might vary. The reliance on proprietary or large pretrained models (Qwen3-Omni, Wan2.2, HunyuanVideo-Foley) for initialization and distillation is noted, which aids reproducibility for those with access to these resources. The code is not explicitly linked in the text, but the methodology is described with sufficient detail for replication.
The primary limitation is that the alignment is learned during training only; the model does not enforce a shared latent distribution, meaning the latents are still modality-specific but aligned in a shared embedding space via the contrastive head. This might limit direct cross-modal generation without additional mapping layers. The sync probing metric, while useful, is a proxy and may not perfectly correlate with human perception of synchronization in complex scenes. The dataset, while large, is web-scraped and may contain noise despite filtering. The reliance on Qwen3-Omni for distillation assumes the availability of this specific large model, which might not be accessible to all researchers.
OmniVAE contributes to the field of unified multimodal generation by providing a stronger foundational representation for audio-video tasks. By decoupling the alignment learning from the generative model, it allows downstream generators to focus on content creation rather than learning synchronization from scratch. This could accelerate the development of high-quality, synchronized audio-video generative models for entertainment, education, and simulation. The emphasis on fine-grained temporal alignment addresses a key challenge in current generative AI, potentially leading to more realistic and usable synthetic media. OmniVAE introduces a novel jointly trained audio-video VAE with segment-level contrastive alignment and semantic distillation, significantly improving cross-modal synchronization and generation quality in text-to-audio-video tasks.
Joint Embedding Predictive Architectures (JEPA) have recently emerged as a paradigm for learning world models by predicting latent representations, offering a promising direction for self-supervised learning. While initial attempts have applied JEPA to the music domain, it remains unclear how such frameworks can naturally support the formation of a world model for music. In this work, we propose to learn a world model of piano sound using JEPA by framing music as an action-conditioned system: the audio is treated as the state, and the pianoroll as the instrument action. Given a current audio state and an action, the model predicts the resulting future audio state, mirroring how humans learn musical sound through interaction. The model is trained in a fully offline setting using paired audio-pianoroll data, without environment interaction. Experiments show that the learned model captures the relationships between musical actions and their resulting sound. The resulting representations support downstream tasks, including beat tracking, composer identification, and key estimation, and enable piano transcription via planning, by searching for actions that best explain a target sound.
Primary: McGill University
All Institutions: McGill University, Advanced Machine Intelligence (AMI Labs), Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT), Mohamed bin Zayid University of Artificial Intelligence (MBZUAI), Schulich School of Music
[One sentence main contribution]. The paper introduces Music-JEPA, an action-conditioned world model that learns latent dynamics of piano music by predicting future audio states from current states and pianoroll actions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This work makes a significant contribution to self-supervised learning in audio by moving beyond passive representation learning to active, action-conditioned modeling. The technical execution is solid, with clear demonstrations of learned dynamics and competitive performance on downstream tasks. While the novelty is incremental (applying known JEPA techniques to a new domain with a specific conditioning mechanism), the significance lies in the rigorous evaluation of action-conditioned latent spaces for music, providing a strong baseline for future research in controllable music generation and understanding. The limitation of requiring paired data is a notable constraint, but the results validate the potential of this paradigm.
The paper proposes Music-JEPA, an action-conditioned Joint Embedding Predictive Architecture for piano music. The core innovation lies in framing music as a dynamical system where audio spectrograms are states and pianoroll/pedal signals are actions. The model predicts future latent states given current states and actions, leveraging the EMA-based training scheme to prevent collapse. The methodology is technically sound, adapting recent vision JEPA techniques to the audio domain with a specific focus on action-conditioning. The use of ViT backbones for both spectrogram and pianoroll inputs is standard but effective. The introduction of an inverse predictor for transcription via planning is a notable conceptual extension, although the implementation relies on amortized optimization rather than direct latent-space planning, which is a pragmatic choice given the high dimensionality of the action space.
The experimental evaluation is comprehensive, covering temporal dynamics analysis, downstream MIR tasks (beat tracking, composer identification, key recognition), and transcription via planning. The results show that Music-JEPA outperforms a passive audio-only JEPA baseline and remains competitive with MERT, a much larger model, on several tasks. The ablation studies on perturbation sensitivity provide strong evidence that the model learns meaningful action-conditioned dynamics rather than just temporal proximity. The transcription results are mixed; while pedal estimation is strong, note-level transcription lags behind supervised methods, which is expected for a generative/planning approach but highlights current limitations. The comparison with MERT is fair and demonstrates the efficiency of the proposed approach.
The paper provides detailed hyperparameters, dataset splits (MAESTRO v3.0.0), and architectural details. The training algorithm is clearly outlined. However, the code and pretrained models are stated to be "released soon," which currently hinders immediate reproducibility. The specific patch sizes and embedding dimensions are provided, allowing for architectural replication.
The primary limitation is the reliance on paired audio-MIDI data, which restricts the model to domains with precise action annotations (like piano) and prevents direct application to unannotated audio. The transcription performance, particularly for notes, is inferior to supervised methods, indicating that the latent space may not be perfectly invertible or that the planning objective is not yet optimized for high-fidelity reconstruction. The model's ability to generalize to other instruments or genres is untested. Additionally, the "planning" aspect is implemented via an inverse model (amortized planning) rather than search-based planning, which limits the interpretability of the "world model" aspect in terms of explicit trajectory optimization.
This work contributes to the growing field of world models for audio and music information retrieval. By demonstrating that action-conditioned latent dynamics can capture musical structure, it opens avenues for more interactive and controllable music AI systems. The approach could inspire similar frameworks for other domains where actions influence observations, such as robotics or speech synthesis. The focus on self-supervised learning with minimal labeled data aligns with broader trends in efficient AI. [One sentence main contribution]. The paper introduces Music-JEPA, an action-conditioned world model that learns latent dynamics of piano music by predicting future audio states from current states and pianoroll actions. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. This work makes a significant contribution to self-supervised learning in audio by moving beyond passive representation learning to active, action-conditioned modeling. The technical execution is solid, with clear demonstrations of learned dynamics and competitive performance on downstream tasks. While the novelty is incremental (applying known JEPA techniques to a new domain with a specific conditioning mechanism), the significance lies in the rigorous evaluation of action-conditioned latent spaces for music, providing a strong baseline for future research in controllable music generation and understanding. The limitation of requiring paired data is a notable constraint, but the results validate the potential of this paradigm.
Real-time score following from sheet images remains chal- lenging because the model must process streaming au- dio while resolving highly repetitive visual patterns un- der strict latency constraints. Recent image-based meth- ods have attempted to use multi-resolution prediction by simultaneously predicting the positions of the active sys- tem, bar, and note. However, their predictions across these different levels of notation are independent, which makes the predictions unstable and introduces unnecessary ex- tra search space for bar- and note-level predictions. Most existing methods also lack mechanisms to recover from score discontinuities, such as repeats, da capo (D.C.), or coda jumps. This paper proposes CODA, to the best of our knowledge, the first real-time score following system that addresses both gaps. CODA explicitly exploits the cascaded structure of music scores: it first selects the ac- tive system, then the active bar within it, and finally the active note within the selected bar. This enforces pre- diction consistency across resolutions. A silence-driven break mode enables recovery from arbitrary score discon- tinuities without requiring knowledge of the repeat struc- ture. Evaluated on the Multimodal Sheet Music Dataset (MSMD) piano benchmarks, CODA achieves state-of-the- art tracking accuracy and discontinuity-recovery perfor- mance under real-time throughput. Code is available at https://github.com/ValleyC/CODA.
Primary: University of Alberta
All Institutions: University of Alberta, Western University
This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
The paper proposes CODA, a cascaded online alignment framework for real-time image-based score following. The core methodological innovation is the reformulation of the tracking problem from independent multi-resolution object detection (as seen in prior work like CYOLO-SB) to a cascaded selection-and-regression task. By explicitly predicting the system, then the bar within that system, and finally the note within that bar, the model enforces geometric consistency and significantly reduces the search space at each stage. The architecture utilizes a causal Mamba encoder for audio processing, which efficiently handles streaming data with linear complexity, paired with a CNN visual backbone. A key technical contribution is the "silence-driven break mode," which leverages the observation that score discontinuities (repeats, da capo, coda jumps) are often preceded by silence. This allows the system to reset its temporal priors and re-localize without requiring explicit symbolic knowledge of the score's repeat structure. The use of beam search with learned temporal priors further stabilizes the predictions.
The evaluation is conducted on the Multimodal Sheet Music Dataset (MSMD), a standard benchmark for this task. The authors report state-of-the-art results in tracking accuracy on both synthetic and real-audio settings compared to strong baselines like CYOLO-SB. Crucially, they introduce a new "repeat-aware jump test benchmark" with manually annotated repeat structures for the 94 test pieces, addressing a significant gap in the literature where discontinuity handling was previously unstandardized. The results show substantial improvements in system and bar accuracy, and notably, CODA demonstrates robust recovery from jumps, whereas baselines fail. The ablation studies effectively isolate the contributions of the cascade structure, beam search, and cross-attention mechanisms.
The paper provides detailed implementation details, including audio preprocessing (log-filterbank, hop size), model architecture dimensions, training schedules (two-phase curriculum with scheduled sampling), and hyperparameters. The code is made publicly available on GitHub. The dataset (MSMD) is publicly accessible. The inclusion of a specific benchmark for jump recovery enhances reproducibility for future work in this niche area.
The method is evaluated exclusively on solo piano music from the MSMD dataset. The performance on more complex musical textures (chamber music, orchestral) or different instruments is not reported. The reliance on synthesized audio for the primary evaluation setting (Setting I) may not fully capture the challenges of real-world acoustic recordings, although Setting II provides some real-audio validation. The "silence" assumption for jump recovery, while effective for many classical pieces, might fail in continuous musical passages where performers execute jumps without significant pauses.
This work advances the field of Human-Computer Interaction (HCI) in music, specifically interactive accompaniment and automated page-turning systems. By enabling robust real-time score following from images without requiring symbolic scores, it lowers the barrier to entry for using digital sheet music in performance settings. The standardized benchmark for discontinuity handling will facilitate fairer comparisons in future research. This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
Omni models transcribe clean, single-speaker speech well, but their accuracy drops sharply when speakers overlap and the scene is noisy, exactly where knowing who said what matters most. A natural fix is a short scene description. We show why this is risky: answer-bearing text lets the model copy instead of listen, so the score rises although nothing has been heard; a silent test exposes this shortcut at once. We call this failure mode perception bypass and address it with Audio-Grounded Scaffold Context (AGSC). AGSC links three steps: first, we build clues from audio to guide listening without giving the answer; second, answer-overlap and silence tests probe them for leakage and audio dependence; finally, those clues scaffold training but vanish at test time, yielding no-clue capability. Across three heterogeneous Omni models, training on AGSC lowers no-clue capped mean permutation word error rate (mpWER) on overlapping, noisy speech from 25%-71% to 9%-15%. For streaming control, we formulate a joint GDPO task in which the model learns when to use a clue and how to produce a speaker-attributed transcript from separately normalized format, gate, and transcript rewards. After internalization, AGSC adds almost no inference overhead.
Primary: unknown
All Institutions: unknown
This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
The paper proposes Audio-Grounded Scaffold Context (AGSC), a framework designed to mitigate "perception bypass" in Omni models—where models rely on textual context rather than audio signals. The methodology involves synthesizing "clues" (partial, shuffled words, time windows) from audio using off-the-shelf front-ends (diarization, ASR, VAD). These clues are used during Supervised Fine-Tuning (SFT) and Group Reward-Decoupled Policy Optimization (GDPO). The core innovation lies in the training protocol: using a "silent test" (silencing the audio input) to detect if the model is copying text, and ensuring that performance gains persist even when the scaffold clues are removed at inference time (internalization). The GDPO component introduces a novel reward structure with separate normalization for format, gate, and transcript rewards, and a conditional reward mechanism to prevent gate collapse. While the concept of using scaffolds is not entirely new, the specific application to preventing perception bypass in Omni models and the rigorous "silent test" evaluation metric constitute a meaningful methodological contribution to robust multimodal training.
The authors evaluate their approach on three heterogeneous Omni models (Qwen3-Omni, MiniCPM-o, Ming-flash-omni) using a custom benchmark, Context-Speech Bench (CSB), which includes overlapping and noisy speech scenarios. The results show significant improvements in mean permutation Word Error Rate (mpWER) for the smaller models (Ming, MiniCPM) after SFT, reducing errors from ~25-68% to ~8-15%. The paper provides a thorough ablation study, including the "silent test" to prove that gains are not due to text copying. The GDPO experiments demonstrate the ability to learn gating strategies, although the full-chain joint training shows some instability (gate collapse) which is addressed with a secondary supervised step. The evaluation is comprehensive, covering both offline transcription and streaming control, and the use of multiple models strengthens the generalizability claims.
The paper states that code, benchmark manifests, and result evidence will be open-sourced upon acceptance. The methodology describes the pipeline components (Pyannote, Silero, Sepformer) and the training hyperparameters (LoRA ranks, learning rates). However, the reliance on specific versions of large base models (Qwen3, MiniCPM, Ming) and the complex synthesis pipeline for clues may pose reproducibility challenges. The description of the GDPO reward normalization and the specific "silent test" protocol is detailed enough to allow replication by researchers familiar with RLHF techniques.
The paper acknowledges that the external detector for gating has lower performance than the learned gate but notes the learned gate can suffer from collapse. The improvements are less pronounced for the strongest baseline model (Qwen3), suggesting diminishing returns for already high-performing systems. The "clue" generation relies on imperfect off-the-shelf tools, which might introduce noise or errors that the model must learn to ignore, potentially limiting the ceiling of performance. Furthermore, the "silent test" is an aggregate control and may not catch all forms of shortcut learning. The paper also notes that unconditional clue usage at test time can hurt performance, highlighting the necessity of the learned gate.
This work addresses a critical failure mode in multimodal AI systems, enhancing their reliability in real-world scenarios like meetings and call centers where audio is often noisy or overlapping. By preventing "perception bypass," the method ensures that models are genuinely listening, which is crucial for trust and safety in automated transcription and assistance systems. The Context-Speech Bench provides a valuable resource for evaluating robustness in challenging acoustic conditions. This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks. Yet the same detector that achieves less than 1% error on one dataset can see its error rate increase twentyfold when evaluated on a different dataset. We argue that one contributing factor is speaker-identity reliance: standard training corpora correlate speaker identity with the genuine/synthetic label, allowing detectors to partially rely on speaker-related cues rather than synthesis artifacts alone. We propose the Identity Sensitivity Score (ISS), a per-utterance diagnostic that quantifies how much a detector's output changes across different speaker identity contexts. ISS requires no ground-truth labels at inference time and can be computed from the detector score and a pool of reference speaker examples. Across two detectors and two datasets, incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified utterances, and ISS alone predicts misclassification with area-under-curve (AUC) up to 0.954. To test whether ISS actually captures identity-sensitive behavior rather than serving only as a proxy for prediction confidence, we apply voice conversion to 500 utterances and measure the resulting detector-score shift. Utterances flagged as identity-sensitive by ISS respond 19 to 30 times more strongly to this manipulation than utterances flagged as stable. These results position ISS as a practical inference-time diagnostic for speaker-dependent failure analysis in audio deepfake detection.
Primary: Michigan State University
All Institutions: Michigan State University
The paper introduces the Identity Sensitivity Score (ISS), a label-free diagnostic metric that quantifies speaker-identity reliance in audio deepfake detectors, demonstrating that identity-sensitive predictions are strongly correlated with misclassification and distributional shift failures.
The paper proposes the Identity Sensitivity Score (ISS), a novel diagnostic metric designed to quantify the extent to which an audio deepfake detector relies on speaker identity cues rather than synthesis-specific artifacts. The methodology is theoretically grounded in the sensitivity of sigmoid decision boundaries; it injects speaker identity context via logit-space perturbation using external ECAPA-TDNN embeddings. This approach is clever because it is label-free at inference time, requiring only the detector's output and a pool of reference speaker embeddings. The method avoids the computational cost of generating new audio samples for probing, instead manipulating the internal representation space. The theoretical motivation linking ISS to decision-boundary proximity provides a strong mathematical justification for why high-ISS scores correlate with misclassification, distinguishing it from standard uncertainty measures like entropy or margin which are monotonic functions of the prediction score itself.
The experimental evaluation is rigorous and well-structured. The authors evaluate ISS across two architecturally distinct detectors (AASIST and RawNet2) and two datasets (ASVspoof 2019 LA and 2021 LA), demonstrating the metric's robustness across different model families. The core finding—that incorrectly classified utterances have ISS scores 29 to 52 times higher than correctly classified ones—is compelling. The use of Voice Conversion (FreeVC) as a controlled intervention to validate that ISS captures genuine identity sensitivity rather than just boundary proximity is a strong empirical contribution. The statistical significance is well-established with Mann-Whitney U tests and Cohen's d. The analysis of attack types (TTS vs. VC vs. Hybrid) adds depth, revealing that different failure modes (identity-driven vs. artifact-driven) can be distinguished by ISS, a nuance lost in aggregate metrics.
The paper provides sufficient detail for reproduction. The datasets (ASVspoof 2019/2021 LA) are standard benchmarks. The detectors (AASIST, RawNet2) are widely available. The external speaker model (ECAPA-TDNN) is also standard. The specific configuration parameters ($K=10$, $\lambda=5.0$) are provided. However, the paper notes that $\lambda$ tuning requires a small labeled development set, which is a minor practical hurdle but does not prevent reproduction. The code is not explicitly linked in the provided text, but the methodology is clear enough that implementation is straightforward.
The authors correctly identify several limitations. ISS relies on an external speaker embedding model, meaning it measures sensitivity to identity as represented in ECAPA-TDNN's space, which may not perfectly align with the deepfake detector's internal identity representations. The method requires a small labeled development set for hyperparameter tuning ($\lambda$), although this is a one-time offline step. The evaluation is limited to two detectors and two datasets; extending to SSL-based detectors (e.g., Wav2Vec2) is noted as future work. Furthermore, while ISS diagnoses the *cause* of potential failure (identity reliance), it does not inherently correct it; training interventions to mitigate this reliance are outside the scope of this work.
This work has significant implications for the reliability and trustworthiness of audio deepfake detection systems, which are critical for security, forensics, and media integrity. By providing a tool to diagnose *why* a detector fails (identity bias vs. artifact ambiguity), ISS enables more targeted improvements in model training and deployment. It highlights a systemic issue in current benchmarks where speaker identity correlates with labels, potentially leading to over-optimistic performance estimates. This diagnostic capability can help researchers build more robust, generalizable detectors that rely on genuine synthesis artifacts rather than spurious correlations. The paper introduces the Identity Sensitivity Score (ISS), a label-free diagnostic metric that quantifies speaker-identity reliance in audio deepfake detectors, demonstrating that identity-sensitive predictions are strongly correlated with misclassification and distributional shift failures.
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF < 1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.
Primary: Microsoft Research
All Institutions: Microsoft Research, Shanghai Jiao Tong University, Fudan University, University of Chinese Academy of Sciences
This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
The paper proposes a heterogeneous quantization strategy for the VibeVoice-ASR architecture, specifically targeting edge CPU deployment. The core technical contribution lies in the distinct treatment of the two main components: the VAE tokenizer is quantized to full-pipeline INT8 (I8_S) to address its activation-bound nature, while the autoregressive language model (LM) decoder utilizes BitNet-style ternary weights (I2_S) to address its weight-bound nature during decoding. The authors introduce a progressive quantization-aware training (QAT) schedule using a blending parameter to stabilize training, which is a practical engineering contribution. The inference engine is built on ggml with custom SIMD kernels (AVX2/NEON) for efficient INT8 and ternary (unpacking to INT8 for MAD operations) computation. The methodology is sound and well-reasoned, leveraging established techniques (BitNet, ggml, INT8 quantization) in a novel combination specific to the VAE-LM ASR paradigm.
The evaluation is conducted on a server-grade AMD EPYC CPU, which serves as a proxy for high-end edge hardware, though real-world edge devices (e.g., mobile SoCs) may have different memory bandwidth and cache characteristics. The paper reports a 2.9x compression ratio (4.62 GB to 1.58 GB) and achieves Real-Time Factor (RTF) < 1 with only 3 threads, demonstrating significant efficiency gains over FP16 baselines and competitive performance against Whisper.cpp. Accuracy is evaluated on multiple multilingual benchmarks (MLC, AISHELL, etc.), showing modest degradation (1-4%) compared to the larger FP16 baseline. The comparison with Whisper.cpp is relevant and highlights the benefits of the specific heterogeneous quantization approach for LLM-based ASR.
The paper provides a GitHub link to the codebase (VibeASR.cpp) and Hugging Face links to the models. The methodology describes the quantization schemes, training schedules, and kernel optimizations in sufficient detail for replication by experts in the field. The use of standard frameworks (ggml) and well-defined quantization protocols enhances reproducibility.
The primary limitation is the lack of streaming support, as noted by the authors. The current implementation is batch-oriented, which limits its applicability in real-time interactive scenarios where low latency is critical. Additionally, the evaluation is performed on a specific server CPU; performance on actual resource-constrained edge devices (ARM-based mobile chips, etc.) is not explicitly detailed, although ARM support is mentioned. The generalizability of the heterogeneous strategy to other VAE-LM architectures (e.g., Whisper) is also unexplored.
This work significantly advances the field of on-device AI by demonstrating that large, accurate LLM-based ASR systems can be deployed on commodity CPUs without GPUs. This has profound implications for privacy-preserving speech recognition, reduced cloud infrastructure costs, and enabling advanced voice assistants on low-power devices. It bridges the gap between high-accuracy LLM-based ASR and practical, efficient edge deployment. This paper presents a highly effective engineering solution for deploying large LLM-based ASR models on edge CPUs through heterogeneous quantization (INT8 for VAE, ternary for LM) and custom SIMD optimization, achieving real-time performance with minimal accuracy loss and significant compression.
Transformers with global attention capture long-range dependencies but can miss the fine-grained local continuity crucial for speech separation. We propose TF-MossFormer, a time-frequency transformer that combines local and global attention to jointly model short- and long-range contexts for monaural speech separation. At its core is a content-aware sliding-window attention mechanism that dynamically adapts receptive fields for stronger local interactions, avoiding the rigidity of static convolutions. Unlike time-domain chunk-based methods, TF-MossFormer leverages the 2D spectrogram to model structure along both time and frequency axes. Convolutional gating between attention layers further improves feature selection and information flow. TF-MossFormer achieves SI-SDRi of 22.6, 24.0, and 24.4 dB on WSJ0-2Mix with 5.9M, 16.9M, and 25.4M parameters, respectively, outperforming prior approaches.
Primary: Alibaba Group
All Institutions: Alibaba Group, Token Foundry
TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
The paper proposes TF-MossFormer, a time-frequency domain architecture for monaural speech separation. The core innovation lies in integrating a content-aware sliding-window local attention mechanism with global self-attention, augmented by convolutional gating. This hybrid approach aims to balance the computational efficiency and local feature preservation of sliding windows with the long-range dependency modeling of global attention. The architecture follows a dual-path (time-frequency) design, processing spectro-temporal features through stacked blocks. While the combination of local and global attention is not entirely new in the broader transformer literature (e.g., Longformer, BigBird), its specific adaptation to the 2D spectrogram structure with convolutional gating for speech separation is a targeted and reasonable engineering contribution. The use of Conv-SwiGLU and RMSGroupNorm indicates a solid understanding of modern transformer stabilization techniques.
The experimental evaluation is conducted on the standard WSJ0-2Mix benchmark. The authors report results for three model scales (Small, Medium, Large) with parameter counts ranging from 5.9M to 25.4M. The results demonstrate state-of-the-art performance (SI-SDRi up to 24.4 dB) compared to prior works like TF-LocoFormer, TF-GridNet, and SepTDA2. The ablation studies provide valuable insights into the hyperparameters (window sizes) and the architectural choices (order of local/global attention, presence of gating). The comparison with SPMamba highlights efficiency gains. However, the evaluation is limited to a single dataset (WSJ0-2Mix), which restricts the generalizability claims. The computational complexity (MACs) is reported, showing competitive efficiency, which is a strong point.
The paper provides sufficient implementation details, including STFT parameters (16ms window, 8ms hop), optimizer settings (AdamW, learning rate schedule), and loss function (SI-SDR). The architecture diagrams and block descriptions are clear. The use of the ESPnet pipeline is mentioned, which aids reproducibility for those familiar with the framework. However, the specific code repository is not linked in the text provided, and some hyperparameters for the ablation studies (e.g., exact dimension $D$ for the medium/large models) are referenced but not explicitly detailed in the text snippets provided (though implied by parameter counts).
The primary limitation is the lack of evaluation on other benchmarks (e.g., LibriMix, WHAM!, or real-world noisy mixtures), which limits the assessment of robustness. The "content-aware" nature of the sliding window is mentioned but the mechanism for dynamic adaptation is not deeply elaborated in the provided text, raising questions about whether it is truly dynamic or just a fixed window with a specific size. The focus on WSJ0-2Mix, while standard, does not demonstrate performance on more challenging, diverse datasets.
This work contributes to the field of audio signal processing, specifically speech separation, which has applications in hearing aids, voice assistants, and audio post-production. By improving the efficiency and quality of TF-domain separation, it supports the development of more accessible and effective speech processing tools. The hybrid attention mechanism could inspire similar architectures in other sequential data domains. TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning. Code, demos, and listening-test results are available at https://haozhang6720.github.io/SoundscapeAgentDemoPage/.
Primary: Tencent Hunyuan
All Institutions: Wuhan University, Tencent Hunyuan
SoundscapeAgent presents a structured, agentic approach to soundscape synthesis that enables controllable generation and provides valuable structured supervision for audio-language reasoning tasks. The paper demonstrates that explicit scene planning and hybrid asset acquisition can produce high-quality, multi-event soundscapes that outperform single-shot baselines in complex scenarios and effectively augment training data for downstream audio reasoning models.
The paper proposes SoundscapeAgent, an agentic framework that decomposes soundscape synthesis into explicit planning, asset acquisition, and rendering stages. The core innovation lies in using an LLM to generate a structured scene plan (events, roles, timing, gain) which is then executed by deterministic rendering primitives. This contrasts with end-to-end text-to-audio models by offering inspectability and controllability. The methodology combines retrieval from a curated asset library with on-demand generation for missing assets, followed by a timeline-based mixer. While the concept of "agentic" audio is emerging, the specific implementation here is a composite of existing techniques (LLM planning + T2A generation + audio mixing) rather than a novel algorithmic breakthrough. The novelty is moderate, primarily in the system-level integration and the specific focus on structured supervision for downstream reasoning.
The evaluation is split into two tracks. Track A compares generation quality against baselines (TangoFlux, EzAudio, AudioLDM2) using human listening tests and CLAP scores. The results show competitive performance, particularly in complex, multi-event, or affective scenes, validating the controllability aspect. Track B evaluates the utility of the generated data for training audio-language models on the MMAU benchmark. The results demonstrate that adding agent-generated data improves reasoning accuracy, especially in sound and music domains, over real-only baselines. This is a strong empirical result supporting the value of structured synthetic data. However, the downstream model uses a frozen encoder and only trains a small projector, which limits the assessment of the data's full potential compared to fine-tuning larger models. The evaluation is solid but relies on standard benchmarks and metrics.
The paper provides a demo page with code links (implied by the URL, though the specific GitHub repo isn't explicitly named in the text, the URL points to a project page). The methodology is described in detail, including the asset filtering criteria (CLAP score, production complexity) and the rendering pipeline. The use of open-source models (Qwen, Whisper, CLAP) aids reproducibility. However, the "private in-house data" mentioned in the asset library is not accessible, which may limit the exact replication of the asset acquisition phase. The prompt engineering for the LLM agent is not fully detailed, which could affect reproducibility of the scene plans.
The authors acknowledge limitations including dependence on asset inventory quality, the domain gap between synthetic and real audio, and the bias towards non-speech events in the asset library. The evaluation of speech capabilities is limited. The reliance on an LLM for planning introduces potential latency and error propagation if the plan is flawed. The "offline prior mode" improves throughput but may reduce the adaptability of the generation to specific user intents compared to online planning.
This work contributes to the field of audio-language modeling by providing a scalable method for generating structured, high-quality synthetic data for training. It addresses the need for fine-grained temporal and compositional supervision in audio understanding. The agentic approach could inspire similar frameworks in other multimodal domains. However, the generation of synthetic audio data raises concerns about data contamination and the potential for propagating biases present in the underlying models or asset libraries. SoundscapeAgent presents a structured, agentic approach to soundscape synthesis that enables controllable generation and provides valuable structured supervision for audio-language reasoning tasks. The paper demonstrates that explicit scene planning and hybrid asset acquisition can produce high-quality, multi-event soundscapes that outperform single-shot baselines in complex scenarios and effectively augment training data for downstream audio reasoning models.
In this work, we propose an interpretable framework for speech deepfake detection based on artifact-specific expert models. Rather than relying on black-box decisions, the framework provides human-understandable evidence, which is critical in high-stakes settings. Each expert is trained to detect a specific speech synthesis artifact, and its output is calibrated into a log-likelihood ratio that serves as an interpretable evidence score. We evaluate five artifact-specific experts and show that, with proper calibration, they can capture their target artifacts and produce meaningful evidence. Importantly, each expert estimates only the presence of its assigned artifact rather than directly performing the final decision. Their outputs are aggregated into an ensemble to produce the actual real-versus-fake classification, while maintaining interpretability by indicating how strongly each expert supports or contradicts a fake classification. Results show that artifact-specific experts capture interpretable signals of synthetic speech across multiple generation pipelines.
Primary: Politecnico di Milano
All Institutions: Politecnico di Milano, National Institute of Informatics
This paper presents a novel, interpretable framework for speech deepfake detection by decomposing the problem into artifact-specific experts trained on pseudo-fake data and aggregating their calibrated log-likelihood ratios, offering a transparent alternative to black-box models in forensic contexts.
The paper proposes a modular, interpretable framework for speech deepfake detection that decomposes the detection task into five specific synthesis artifacts: F0 inconsistencies, oversmoothing, degraded phase, energy inconsistencies, and tonal artifacts. The core methodological innovation lies in the use of "pseudo-fake" data generation—applying controlled, algorithmic manipulations to real speech to train artifact-specific expert models. This avoids the confounding factors present in real deepfake data. The outputs of these experts are calibrated into Log-Likelihood Ratios (LLRs) using either Kernel Density Estimation (KDE) or Constrained Maximum-Likelihood Gaussian (CMLG) methods, allowing for a principled, forensic-style aggregation of evidence. The approach is grounded in signal processing principles rather than purely end-to-end deep learning, aiming for intrinsic interpretability.
The authors evaluate their framework on ASVspoof 2019, ASVspoof 5, and SpoofCeleb datasets. They demonstrate that individual experts can detect their target artifacts with above-chance performance, though not state-of-the-art accuracy. The ensemble of calibrated experts achieves competitive results, with the sum-aggregation strategy outperforming max and learned gating mechanisms. The interpretability is validated by showing that experts activate strongly for attacks known to exhibit those specific artifacts (e.g., the phase expert for Neural Source-Filter vocoders). The paper also includes an ablation study on calibration data composition, finding that a pooled calibration set across experts yields the most stable LLRs.
The paper provides detailed descriptions of the pseudo-fake generation pipelines, including specific parameters for manipulations (e.g., Gaussian filter standard deviation, F0 scaling factors). The feature extraction methods (pYIN, LSF, STFT) and model architectures (MLPs, LCNN) are specified. However, the exact code for the pseudo-fake generation and the specific random seeds for the experiments are not provided in the text, which may hinder exact reproduction. The reliance on external libraries (e.g., pYIN) is noted but implementation details are standard.
The primary limitation is the reliance on predefined artifacts. The framework cannot detect novel or unknown artifacts that do not fit the five modeled categories. The performance is lower than state-of-the-art black-box detectors, which is acknowledged but remains a practical constraint. The assumption of independence among artifact features for LLR summation is a simplification that may not hold perfectly in complex synthesis pipelines. Furthermore, the "pseudo-fake" approach may not fully capture the complexity and distribution of artifacts found in real-world, multi-stage deepfake generation systems.
This work contributes to the critical field of audio forensics and security by providing a transparent method for deepfake detection. The interpretable nature of the system makes it suitable for high-stakes applications where explaining the decision to a human judge or jury is required. It shifts the paradigm from opaque black-box detectors to evidence-based reasoning, potentially increasing trust in automated detection systems. This paper presents a novel, interpretable framework for speech deepfake detection by decomposing the problem into artifact-specific experts trained on pseudo-fake data and aggregating their calibrated log-likelihood ratios, offering a transparent alternative to black-box models in forensic contexts.
Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing Academy of Artificial Intelligence, Institute of Computing Technology, Chinese Academy of Sciences, University of California, Merced, The University of Queensland
[One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
The paper proposes Audio-Zero, a label-free self-evolution framework for Large Audio Language Models (LALMs). The core innovation is an "auditory self-play game" that converts unlabeled audio contrast pairs into verifiable training signals. The method involves two coupled stages: a listening stage where the model generates descriptive clues for multiple "players" (some hearing a reference, one hearing a variant), and an attribution stage where the model acts as a judge to identify the "odd listener" based on inconsistencies in the clues. The reward signal is intrinsic to the game mechanics (correct identification) and a rule-based content utility score for the clues. The optimization uses Group Relative Policy Optimization (GRPO). The approach is theoretically sound and addresses a genuine gap in the field: the lack of fine-grained, label-free supervision for audio reasoning. The mechanism of using the "odd-one-out" task to force fine-grained perceptual description is clever and aligns well with recent trends in self-play and self-evolution for LLMs and VLMs.
The authors evaluate Audio-Zero on three benchmarks: TREA (temporal reasoning), MMAU Test-mini (general audio understanding), and MMAR (deep audio reasoning). Experiments are conducted on two strong backbones: Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B. The results show consistent improvements over the base policies and several strong baselines (both label-dependent and label-free). Notably, the gains are most pronounced on TREA, the fine-grained temporal reasoning benchmark, which supports the paper's claim of improving fine-grained perception. The ablation studies effectively demonstrate the necessity of each component (listening stage, attribution stage, vote-aware feedback). The self-evolution dynamics analysis (tracking win rates, clue length, and benchmark scores over iterations) provides compelling evidence that the model is indeed learning and evolving rather than just overfitting to a static reward. The use of only 2k unlabeled pairs is a significant efficiency advantage.
The paper provides detailed implementation details, including hyperparameters (learning rate, KL coefficient, number of players/rounds), dataset construction (filtering 2k pairs from Audio-Alpaca), and prompt templates. The GRPO implementation details are standard but well-specified. The diagnostic metrics for analyzing clue quality are clearly defined. While the code is not explicitly linked in the provided text, the description is sufficient for reproduction by researchers familiar with RLHF/GRPO pipelines. The reliance on rule-based scorers for the listening reward is a potential point of variability, but the rules are described.
The method relies on the availability of audio contrast pairs. While the paper claims to use "unlabeled" pairs, it sources them from Audio-Alpaca, which is a preference dataset. The filtering process to create contrast pairs (ensuring they share semantics but differ in fine-grained details) is not fully detailed, raising questions about the scalability and generalizability of the pair generation process. The rule-based listening reward, while effective, may not capture all nuances of good auditory description compared to a learned reward model. Furthermore, the performance gains, while significant, are still modest in absolute terms on some benchmarks (e.g., MMAU Avg. 60.50 vs 57.50 for Qwen2-Audio), suggesting that self-evolution alone may not solve all reasoning deficits without some form of structured supervision or larger-scale data.
This work contributes to the development of more autonomous and data-efficient AI systems. By reducing reliance on expensive human annotations, it lowers the barrier to training high-capability audio models. The improved fine-grained audio reasoning capabilities could benefit applications in accessibility (e.g., better descriptions for the visually impaired), audio forensics, and interactive AI assistants. The self-play paradigm could be extended to other modalities or tasks where verifiable rewards are hard to construct. [One sentence main contribution]. Audio-Zero introduces a label-free self-evolution framework for LALMs that uses an auditory self-play game to improve fine-grained reasoning without external annotations. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a novel and technically sophisticated approach to enhancing the fine-grained reasoning capabilities of Large Audio Language Models. By formulating the learning process as a self-play game with verifiable intrinsic rewards, Audio-Zero effectively circumvents the bottleneck of scarce fine-grained annotations. The methodology is well-grounded in recent advances in self-evolution and reinforcement learning, and the experimental results demonstrate its efficacy across multiple benchmarks and model architectures. The diagnostic analysis provides valuable insights into how fine-grained auditory descriptions emerge from game pressure. This work is a significant contribution to the field of audio AI, offering a scalable and efficient path to improving model performance.
Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen
SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
The paper proposes SimulS2ST-Omni, a data-efficient framework for long-form streaming Speech-to-Speech Translation (S2ST). The core methodological contributions are threefold: (1) A joint text-code trajectory supervision mechanism that schedules target text and acoustic semantic codes simultaneously, eliminating the need for unstable, separate speech-side emission controllers. (2) A two-stream "Thinker-Talker" architecture that decouples linguistic reasoning (Thinker) from dense acoustic prediction (Talker), mitigating modality interference common in unified decoders. (3) A training recipe leveraging auxiliary multitask supervision (ASR, S2TT, MT, TTS) to enable robust performance with only ~2k hours of paired S2ST data, a significant reduction from prior works requiring tens of thousands of hours. The approach extends trajectory-based simultaneous translation to speech output by using cross-lingual word alignments and monotonicity filtering (NIR) to construct stable read/wait/write trajectories.
The authors conduct comprehensive experiments on offline S2ST (CVSS-T), sentence-level streaming (RealSI), and long-form streaming (RealSI, ACL60/60-dev). They compare against state-of-the-art public baselines (UniSS, Seamless-M4T) and closed-source systems (LiveInterpret 2.0). Results show that the Thinker-Talker architecture significantly outperforms the matched unified-decoder baseline in translation quality (ASR-BLEU, Text-BLEU) while maintaining competitive or superior speech quality metrics (A.PCP, SIM-O). The system matches or exceeds LiveInterpret 2.0 in several latency tiers. Ablation studies confirm the importance of NIR-based trajectory filtering and the architectural decoupling. The evaluation is rigorous, covering multiple metrics and latency constraints.
The paper provides detailed descriptions of the data construction process, including the use of public ASR/S2TT corpora to synthesize paired S2ST data, the alignment tools (SimAlign), and the filtering criteria (NIR, ASR error rates). Training hyperparameters, stages, and model initialization details are provided in the appendix. The use of open-source components (Qwen2.5-Omni, DualCodec, Whisper, Paraformer) enhances reproducibility. However, the exact codebase for the trajectory construction and the specific synthesized data pool are not explicitly linked to a public repository, which may hinder exact replication.
The experiments are limited to Chinese-English directions due to data scarcity. The paired S2ST data is synthesized rather than naturally recorded simultaneous interpretation, which may not fully capture real-world interpreting phenomena (e.g., disfluencies, interruptions). The speech generation backend (chunk-wise Flow Matching) is not fully streaming-native, potentially limiting inter-chunk coherence and inference efficiency. The work does not address duplex interaction or broader multilingual settings.
This work advances the field of real-time cross-lingual communication by demonstrating that high-quality streaming S2ST is achievable with limited paired data. It addresses ethical concerns regarding voice cloning and impersonation by acknowledging the risks and suggesting safeguards. The data-efficient approach lowers the barrier to entry for developing robust S2ST systems. SimulS2ST-Omni presents a significant technical contribution to streaming speech-to-speech translation by introducing a joint text-code trajectory supervision method and a decoupled two-stream architecture that achieves state-of-the-art performance with substantially reduced data requirements, effectively addressing the critical bottleneck of paired S2ST data scarcity.
Existing symbolic music generation models typically use bars as the basic structural unit. However, human perception of musical phrases often does not align with notated bar lines, leading to long-term structural fragmentation. This paper proposes RPPNet-a two-stage deep learning architecture with variable structural boundaries. It first generates variable-length Rhythm-Pitch Primitive (RPP) sequences, where each RPP encodes note count, rhythm, and contour; then decodes the RPP sequences into concrete notes. The grouping of RPPs is automatically derived from acoustic cues, auditory inertia, and similarity perception based on music psychology. Experiments show that melodies generated by RPPNet are superior in both long-term structure and musicality, with significant improvements across all subjective evaluation dimensions. Ablation studies confirm that the performance gain stems from the structural correctness of the psychological representation, rather than from model capacity. This work offers an interdisciplinary perspective for music generation, integrating music theory, computational modeling, and music psychology.
Primary: Zhejiang University
All Institutions: Zhejiang University, ZJU-ZJCM MusicAI Lab, Zhejiang Conservatory of Music
[One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
The paper proposes RPPNet, a hierarchical two-stage architecture for symbolic melody generation. The core innovation lies in replacing fixed bar-level tokenization with perceptually-grouped Rhythm-Pitch Primitives (RPPs). The authors employ a heuristic algorithm based on music psychology (acoustic cues, auditory inertia, similarity perception) to segment melodies into variable-length RPPs. The generation process is decoupled: an RPP-level Transformer generates the structural skeleton, which is then instantiated into note-level events via a time-scale expansion mapping. The methodology is theoretically grounded in cognitive musicology, offering a novel perspective on structural segmentation that challenges the dominant fixed-grid assumptions in models like Museformer and MELONS. The decoupled serial prediction mechanism at the RPP level is a technically sound approach to managing the complexity of long-sequence generation.
The experimental setup utilizes the MelodyNet dataset. The evaluation relies heavily on subjective listening tests (MOS-style) with 15 participants, which is a standard but resource-intensive method for music generation. RPPNet significantly outperforms baselines (Museformer, MELONS) in subjective metrics, particularly in "Structure" and "Overall Impression." The inclusion of ablation studies (Random-Grouped vs. Heuristic-Grouped) is crucial and effectively isolates the contribution of the psychological grouping algorithm from mere variable-length tokenization. However, the objective metrics are limited; the paper notes that PPL and Structural Error are difficult to compare across different tokenization schemes, leading to a reliance on subjective scores. The gap between generated music and human composition (6.69 vs 7.47) remains significant, though the paper attributes this to rhythmic distribution mismatches.
The paper provides a GitHub repository link, which is a positive step for reproducibility. The heuristic algorithm for RPP grouping is described, but the full formalization of the weight function $W()$ is stated to be in the repository, which may contain implementation-specific details not fully explicit in the text. The dataset (MelodyNet) is public. The subjective evaluation protocol is described, but the small sample size (15 participants) limits the statistical power of the findings, although significance tests are provided.
The primary limitation is the reliance on a heuristic, rule-based algorithm for RPP segmentation, which may not generalize well to all musical styles or complex polyphonic textures. The current model is limited to monophonic melody generation. The subjective evaluation, while showing statistical significance, suffers from a small participant pool and potential bias. The objective metrics do not fully capture the perceptual improvements claimed, as standard metrics like PPL are not directly comparable across different tokenization granularities.
This work bridges the gap between computational music generation and music psychology/cognition. By demonstrating that perceptually-grounded structural units lead to more coherent music, it offers a new paradigm for designing generative models that align with human auditory perception. This could influence future research in hierarchical modeling and structured generation beyond music, such as in language or video, where fixed-grid assumptions may also be limiting. [One sentence main contribution]. [RPPNet introduces a perceptually-grounded, variable-length structural representation for symbolic music generation, demonstrating that aligning model boundaries with human auditory grouping principles significantly improves long-term structural coherence compared to fixed bar-level approaches.]
Keyword spotting (KWS) models on embedded devices often need to add new keywords after deployment, but updates are difficult when original training data are unavailable and regressions on existing triggers are unacceptable. At a fixed operating point, our method reduces average new-keyword false reject rate (FRR) from 6.46 to 4.37 versus a parameter-matched separate-model baseline and outperforms parameter-efficient tuning baselines (adapters, LoRA), while using fewer multiply-accumulate operations (MACs) under the same added-parameter budget ($\leq$10k): 16.34M vs 18.45M/20.52M. We achieve this via parameter-capped modular expansion: the base network, including batch-normalization statistics and the core classifier, is frozen, and only a lightweight expansion branch with a separate new-keyword head is trained, preserving core logits, shipped outputs, and thresholds for existing keywords.
Primary: Yandex
All Institutions: Yandex
The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
The paper proposes a "modular network expansion" technique for Keyword Spotting (KWS) on embedded devices. The core innovation is architectural: it freezes the entire base SVDF network (including Batch Normalization statistics) and attaches a lightweight, trainable expansion branch with a separate classifier head for new keywords. This design ensures a strict non-regression guarantee for existing keywords because the core logits remain identical to the pre-deployment model. The method is compared against parameter-efficient fine-tuning (LoRA, Adapters), ensemble methods, and continual learning (EWC). The approach is pragmatic and well-suited for the specific constraints of embedded deployment (no original data, strict latency/parameter budgets). While the concept of "parallel heads" or "expert routing" is not new in the broader ML landscape (e.g., Mixture of Experts, Progressive Neural Networks), its specific application to strict non-regression KWS updates with frozen BN layers is a solid, incremental engineering contribution. It lacks deep theoretical novelty but offers a robust practical solution.
The experiments are conducted on Google Speech Commands v2 (GSC) using five held-out keyword pairs. The evaluation metric is macro False Reject Rate (FRR) at a fixed False Accept Rate (FAR). The results show that the proposed method significantly outperforms the Ensemble baseline (6.46% vs 4.37% FRR) and other parameter-efficient methods (LoRA, Adapters) under a 10k parameter budget. The ablation study on expansion depth is useful, identifying that tapping 4 blocks is optimal. The comparison with Full Fine-tuning highlights the catastrophic forgetting problem, and EWC is shown to be less effective without replay data. The use of Mozilla Common Voice for negative samples is a good practice for realistic FAR calibration. The results are statistically significant (95% confidence intervals provided). However, the evaluation is limited to GSC, which is a relatively clean, controlled dataset. Real-world embedded KWS faces more challenges with noise, channel variation, and long-tail distributions.
The paper provides sufficient detail regarding the architecture (SVDF, BN, Hard-Swish), training setup (Adam, cosine LR, SpecAugment), and evaluation protocol (FAR calibration, 8 runs). The parameter budget (10k) and MAC counts are specified. The code is not explicitly linked, but the description is detailed enough for reproduction. The use of standard datasets (GSC, Common Voice) aids reproducibility.
The primary limitation is the reliance on GSC, which may not reflect the complexity of real-world embedded audio environments. The method assumes that the base model's features are sufficiently generalizable to the new keywords via the expansion branch; if the new keywords are phonetically very distinct from the core set, performance might degrade. The "core-first" decision rule introduces a slight latency overhead (checking core first, then new), although the paper argues this is negligible or manageable. The method is specific to KWS and may not generalize easily to other incremental learning tasks without modification.
This work has significant practical impact for the deployment of always-on voice assistants on resource-constrained devices. By enabling safe, data-efficient updates to keyword inventories, it addresses a major pain point in product lifecycle management for IoT devices. It reduces the need for retraining on large datasets and mitigates the risk of degrading user experience for existing features. The paper presents a practical and effective method for modular expansion of embedded KWS models, achieving strict non-regulation and improved new-keyword detection through a frozen-base, parallel-branch architecture.
The trade-off between robustness, latency, and prosody critically challenges text-to-speech (TTS) systems. Autoregressive models, despite fidelity, are slow and error-prone; non-autoregressive (NAR) alternatives, while fast, often sacrifice prosodic naturalness via rigid alignments. This paper introduces StellarTTS, a novel mobile-optimized NAR TTS framework based on a sparse temporal embedding strategy, enabling granular control of phoneme duration, pronunciation, and prosody. Furthermore, we propose a semantic-aware codec that facilitates efficient single-stage decoding. Conditioned on the sparse temporal embedding, our 83M-parameter lightweight masked generative transformer achieves a real-time factor (RTF) of 0.08. Experiments demonstrate that StellarTTS attains lower latency and stronger robustness compared to state-of-the-art TTS systems, while maintaining competitive performance in audio quality, prosodic naturalness, and speaker similarity.
Primary: Honor Device Co., Ltd.
All Institutions: Honor Device Co., Ltd., Shanghai Jiao Tong University
StellarTTS presents a practical, mobile-optimized NAR TTS system that prioritizes robustness and latency through a novel sparse temporal embedding and semantic-aware codec, achieving significant efficiency gains at the cost of some speaker fidelity.
The paper proposes StellarTTS, a non-autoregressive (NAR) Text-to-Speech (TTS) framework optimized for mobile deployment. The core methodological contributions are twofold: (1) a "semantic-aware codec" that uses Residual Vector Quantization (RVQ) with the 0-th channel distilled from Wav2vec-Bert semantic features, aiming to unify semantic and acoustic token prediction into a single stage; and (2) "sparse temporal embedding," which replaces rigid length regulators with a strategy where phone embeddings are placed only at the center of the duration window, allowing the model to learn prosodic variations while maintaining structural anchors. The architecture is an 83M-parameter masked generative transformer. While the integration of semantic distillation into the codec is a practical engineering choice for latency reduction, the "sparse temporal embedding" is a relatively minor variation of existing alignment-free or soft-alignment techniques in masked generative models (e.g., MaskGCT, F5-TTS). The novelty lies more in the specific system integration for mobile constraints than in fundamental algorithmic innovation.
The experimental evaluation is comprehensive in terms of metrics but reveals a significant trade-off. The model achieves a Real-Time Factor (RTF) of 0.08, demonstrating high efficiency. It shows superior Word Error Rate (WER) on challenging datasets (test-hard), indicating robustness against error propagation common in AR models. However, the speaker similarity (SIM-o) is notably lower than state-of-the-art baselines (e.g., MaskGCT, CosyVoice2). The authors attribute this to the semantic distillation in the codec, which compresses acoustic detail. Subjective evaluations (CMOS/SMOS) show competitive naturalness but do not fully compensate for the objective similarity gap. The ablation studies support the efficacy of the sparse embedding strategy over random anchoring. The comparison with baselines is fair, but the lower SIM-o is a critical limitation for zero-shot TTS applications where speaker fidelity is paramount.
The paper provides sufficient architectural details, including the RVQ structure, loss functions (KL divergence for semantic distillation, MSE for duration prediction, cross-entropy for generation), and training hyperparameters (AdamW, learning rate schedule). The dataset (Emilia) and test sets (Seed-TTS) are standard. However, the specific implementation details of the "semantic-aware codec" training joint optimization and the exact configuration of the duration predictor are somewhat high-level. The code link provided is a demo page, not a GitHub repository, which hinders immediate reproducibility.
The primary limitation is the reduced speaker similarity (SIM-o) compared to SOTA models, which is a direct consequence of the single-stage semantic-aware codec design. The model's performance on very long-form synthesis is not explicitly evaluated, though the NAR nature suggests potential for consistency issues over long contexts. The reliance on a pre-trained semantic encoder (Wav2vec-Bert) for distillation adds an external dependency, though it is frozen during the main generation phase. The "sparse" embedding strategy, while effective for robustness, may struggle with extremely rapid prosodic changes that do not align well with the phone-centric anchor points.
This work contributes to the democratization of high-quality TTS on edge devices, which has significant implications for privacy-preserving voice assistants and low-latency communication tools. By demonstrating that robust, low-latency TTS is possible without sacrificing too much naturalness, it pushes the field towards more practical, deployable AI audio systems. However, the lower speaker similarity might limit its use in high-fidelity cloning applications. StellarTTS presents a practical, mobile-optimized NAR TTS system that prioritizes robustness and latency through a novel sparse temporal embedding and semantic-aware codec, achieving significant efficiency gains at the cost of some speaker fidelity.