Unified audio-visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive media. However, when moving from general audio-video synthesis to music-dance co-generation, the task becomes substantially harder: musical rhythm, phrasing, and accents must drive choreographic motion at fine temporal resolution, and such rhythmic coupling is not captured by unimodal metrics or generic audiovisual consistency scores used in current evaluation practice. We introduce TMD-Bench, a benchmark for text-driven music-dance co-generation that assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The benchmark integrates computable physical metrics with perceptual multimodal judgments, and is supported by a curated rhythm-aligned music-dance dataset and a fine-grained Music Captioner for structured music semantics. TMD-Bench further reveals that (i) modern commercial audio-visual models, such as Veo 3 and Sora 2, produce high-quality music and video, while rhythmic coupling remains less consistently optimized and leaves room for improvement, and (ii) our unified baseline RhyJAM trained on rhythm-aligned data achieves competitive beat-level synchronization while maintaining competitive unimodal fidelity. This presents prospects for building next-generation music-dance models that explicitly optimize rhythmic and kinetic coherence.
Primary: Zhejiang University
All Institutions: Zhejiang University, Tencent, National University of Singapore
The paper presents TMD-Bench, a novel benchmark for music-dance co-generation, and introduces RhyJAM, a unified model that achieves competitive rhythmic alignment and unimodal fidelity. This work significantly advances the evaluation and generation of synchronized audio-visual content, addressing a critical gap in existing methodologies.
The paper introduces TMD-Bench, a comprehensive evaluation framework specifically designed for music-dance co-generation, which is a significant advancement in the field. The methodology is well-structured, integrating both low-level physical metrics and high-level perceptual assessments through a multi-level evaluation pipeline. The authors also develop a unified model, RhyJAM, that generates music and dance in a coherent manner, addressing the critical challenge of rhythmic alignment. The use of a fine-grained Music Captioner for structured music semantics is a novel aspect that enhances the evaluation process.
The experiments are robust, utilizing a curated dataset of 10,000 rhythm-aligned music-dance pairs. The evaluation metrics are comprehensive, covering unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The results demonstrate that RhyJAM outperforms existing models in rhythmic alignment while maintaining competitive unimodal fidelity. The comparative analysis against various baselines, including closed-source and open-source models, provides a clear picture of the model's strengths and weaknesses.
The paper provides detailed implementation details, including training configurations and dataset processing methods, which enhance reproducibility. However, the absence of a publicly available demo or project URL limits the ability for others to directly replicate the results.
One limitation is the reliance on existing commercial models for baseline comparisons, which may not fully represent the capabilities of open-source alternatives. Additionally, while the evaluation framework is comprehensive, the subjective nature of some assessments may introduce variability in human judgments.
The proposed benchmark and model have the potential to significantly impact the fields of generative audio and video synthesis, particularly in applications related to virtual production and interactive media. By addressing the intricate coupling between music and dance, this work paves the way for more sophisticated generative models that can enhance user experiences in entertainment and education. The paper presents TMD-Bench, a novel benchmark for music-dance co-generation, and introduces RhyJAM, a unified model that achieves competitive rhythmic alignment and unimodal fidelity. This work significantly advances the evaluation and generation of synchronized audio-visual content, addressing a critical gap in existing methodologies.
Stem retrieval, the task of matching missing stems to a given audio submix, is a key challenge currently limited by models that discard temporal information. We introduce PHALAR, a contrastive framework achieving a relative accuracy increase of up to $\approx 70\%$ over the state-of-the-art while requiring $<50\%$ of the parameters and a 7$\times$ training speedup. By utilizing a Learned Spectral Pooling layer and a complex-valued head, PHALAR enforces pitch-equivariant and phase-equivariant biases. PHALAR establishes new retrieval state-of-the-art across MoisesDB, Slakh, and ChocoChorales, correlating significantly higher with human coherence judgment than semantic baselines. Finally, zero-shot beat tracking and linear chord probing confirm that PHALAR captures robust musical structures beyond the retrieval task.
Primary: Sapienza University of Rome
All Institutions: Sapienza University of Rome, Moises Systems, Paradigma
PHALAR introduces a novel contrastive framework for musical audio representation that emphasizes phase and pitch equivariance. This work significantly advances the understanding of musical coherence in audio processing, providing a powerful tool for both academic research and practical applications in music technology.
The methodology presented in PHALAR is innovative, leveraging complex-valued neural networks (CVNNs) and a Learned Spectral Pooling layer to address the challenge of musical coherence in audio representation. The paper effectively shifts the paradigm from traditional pooling methods that enforce invariance to a framework that emphasizes equivariance, allowing for the preservation of temporal alignment critical for music tasks. The use of phase information as a geometric representation is a significant advancement, as it directly addresses limitations in existing models that fail to capture rhythmic and harmonic relationships adequately.
The experimental evaluation is robust, demonstrating PHALAR's superiority over state-of-the-art models in stem retrieval tasks across multiple datasets (MoisesDB, Slakh, and ChocoChorales). The paper includes a comprehensive set of experiments, including human-centered evaluations that correlate model outputs with human perception of coherence. The results indicate a significant improvement in retrieval accuracy and alignment with human judgment, validating the effectiveness of the proposed approach.
The authors provide a GitHub repository with code and checkpoints, enhancing the reproducibility of their work. They detail the experimental setup, including dataset construction, training parameters, and evaluation metrics, which allows other researchers to replicate their findings. However, the reliance on specific hardware (NVIDIA A100 GPUs) may pose challenges for some researchers in terms of accessibility.
The paper acknowledges several limitations, including the model's performance degradation in the presence of non-periodic rhythms and tempo drift, which can affect phase coherence. Additionally, the model's training on predominantly Western popular music may limit its applicability to other musical styles. The potential for rigid enforcement of rhythmic standards in automated systems is also noted as a concern.
PHALAR has significant implications for music information retrieval and audio processing, offering a novel approach that could enhance various applications, including automated music production and generative audio models. The framework's emphasis on phase-sensitive data representation could extend to other fields such as radar systems and medical imaging, where preserving complex signal integrity is crucial. PHALAR introduces a novel contrastive framework for musical audio representation that emphasizes phase and pitch equivariance. This work significantly advances the understanding of musical coherence in audio processing, providing a powerful tool for both academic research and practical applications in music technology.
Speech encodes multiple simultaneous attributes--linguistic content, speaker identity, dialect, gender--that conventional single-vector embeddings conflate. We present a factor-partitioned embedding framework that maps each utterance into a single vector whose subspaces correspond to distinct axes of variation. A shared acoustic encoder feeds per-axis linear projection heads, each trained via distillation from a specialist teacher or a contrastive objective over shared-label pairs. The resulting embeddings support attribute-conditioned retrieval: similarity is computed as a signed weighted sum over per-axis cosine scores, allowing retrieval that jointly considers what was said and how --or explicitly suppresses one attribute to surface another. We evaluate on cross-corpus retrieval over corpora sharing the Harvard sentence prompts, demonstrating that signed axis weighting can suppress same-speaker bias and surface semantically matched utterances across recording conditions.
Primary: Department of Speech, Music & Hearing
All Institutions: Department of Speech, Music & Hearing
The main contribution of this paper is the introduction of a novel factor-partitioned embedding framework for speech that allows for controllable multi-axis similarity searches. This work represents a meaningful advancement in the field of speech representation learning, addressing the complexities of speech attributes and providing a robust solution for attribute-conditioned retrieval tasks.
The paper introduces a factor-partitioned embedding framework that effectively separates various attributes of speech into distinct subspaces, allowing for nuanced similarity searches. The methodology is well-structured, employing a shared acoustic encoder and per-axis linear projection heads trained through distillation and contrastive objectives. This approach is innovative in its use of signed axis weighting to control retrieval outcomes, providing a significant advancement over traditional single-vector embeddings that conflate multiple attributes.
The experiments conducted are thorough, utilizing cross-corpus retrieval tasks with well-defined metrics such as Precision@k and preference-flip tests. The evaluation on datasets sharing the Harvard sentence prompts is appropriate, demonstrating the framework's ability to suppress same-speaker bias and surface semantically matched utterances. The results are compelling, showing that the proposed method outperforms baseline models significantly.
The paper provides a clear description of the architecture, data sources, and training processes, which enhances reproducibility. However, the absence of a publicly available implementation or code repository limits the ability for other researchers to replicate the results fully.
One limitation identified is the reliance on specific datasets, which may not generalize well to other speech domains. Additionally, the paper notes challenges in separating the gender axis due to its correlation with speaker identity, suggesting that further exploration is needed to improve this aspect. The potential for mode collapse in models trained without auxiliary tasks is also a concern.
The proposed framework has significant implications for applications in speech retrieval, speaker recognition, and potentially in areas such as voice conversion and personalized speech synthesis. By enabling controllable retrieval based on multiple axes, it opens avenues for more sophisticated user interactions with speech technologies. The main contribution of this paper is the introduction of a novel factor-partitioned embedding framework for speech that allows for controllable multi-axis similarity searches. This work represents a meaningful advancement in the field of speech representation learning, addressing the complexities of speech attributes and providing a robust solution for attribute-conditioned retrieval tasks.
Multimodal Emotion Recognition (MER) has attracted growing attention with the rapid advancement of human-computer interaction. However, different modalities exhibit substantial discrepancies in semantics, quality, and availability, leading to highly heterogeneous modality combinations and posing significant challenges to achieving consistent and reliable emotion understanding. To address this challenge, we propose the Modality-Aware Contrastive and Uncertainty-Regularized (MCUR) framework, which approaches MER from the perspective of representation consistency, aiming to enable robust emotion prediction across heterogeneous modality combinations. MCUR incorporates two core components: (1) Modality Combination-Based and Category-Based Contrastive Learning mechanism (MCB-CL), which encourages samples with the same emotion category and the same available modalities to be close in the representation space; and (2) Sample-wise Uncertainty-Guided Regularization (SUGR), which adaptively assigns sample-wise uncertain weights to samples to optimize training. Extensive experiments demonstrate that MCUR consistently outperforms existing methods, achieving average F1 gains of 2.2% on MOSI, 2.67% on MOSEI, and 4.37% on IEMOCAP.
Primary: University of Electronic Science and Technology of China
All Institutions: University of Electronic Science and Technology of China, Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China
The paper introduces the MCUR framework, which enhances multimodal emotion recognition by promoting representation consistency and addressing uncertainty in predictions. This comprehensive analysis highlights the technical contributions, innovative methodology, and potential impact on the field of machine learning and human-computer interaction.
The proposed MCUR framework presents a novel approach to multimodal emotion recognition (MER) by focusing on representation consistency across heterogeneous modalities. The integration of Modality Combination-Based and Category-Based Contrastive Learning (MCB-CL) and Sample-wise Uncertainty-Guided Regularization (SUGR) is a significant methodological advancement. MCB-CL enhances the discriminative power of representations by enforcing proximity in the embedding space for samples with the same emotion category and modality combination, while SUGR addresses uncertainty in predictions, allowing for adaptive weighting during training. This dual approach is innovative and effectively tackles the challenges posed by modality heterogeneity.
The experiments are comprehensive, utilizing three widely recognized datasets (MOSI, MOSEI, and IEMOCAP) to validate the effectiveness of the MCUR framework. The reported performance improvements over existing state-of-the-art methods, with average F1 gains of 2.2% on MOSI, 2.67% on MOSEI, and 4.37% on IEMOCAP, demonstrate the robustness of the proposed approach. The ablation studies further substantiate the contributions of each component of the framework, revealing the critical role of both MCB-CL and SUGR in enhancing model performance.
The paper provides detailed implementation details, including the training configurations, hyperparameter settings, and evaluation protocols, which are crucial for reproducibility. The authors also mention the use of official implementations for baseline models, ensuring a fair comparison. However, the lack of publicly available code or demo URLs limits the ease of reproduction for external researchers.
While the MCUR framework shows promising results, the paper does not address the potential computational overhead associated with the added complexity of the proposed methods. Additionally, the performance in real-world noisy conditions is not thoroughly evaluated, which could limit the applicability of the framework in practical scenarios. The reliance on specific datasets may also restrict generalizability to other contexts or domains.
The advancements presented in this paper have significant implications for human-computer interaction, particularly in enhancing emotion recognition systems that can adapt to varying modalities. The ability to maintain consistent representations across different modalities can improve the robustness of applications in areas such as virtual assistants, mental health monitoring, and social robotics. The focus on uncertainty in predictions may also lead to more reliable systems that can better handle real-world variability. The paper introduces the MCUR framework, which enhances multimodal emotion recognition by promoting representation consistency and addressing uncertainty in predictions. This comprehensive analysis highlights the technical contributions, innovative methodology, and potential impact on the field of machine learning and human-computer interaction.
Recently, neural directional filtering (NDF) has been introduced as a flexible approach for reconstructing a virtual directional microphone (VDM) with a desired directivity pattern for spatial sound capture. Building on this idea, we propose NDF+, which enables joint neural directional filtering and diffuse sound extraction. NDF+ reformulates VDM estimation into two coupled subtasks: dereverberated VDM reconstruction and diffuse sound extraction. This reformulation enables NDF+ to manipulate diffuse components in the final reconstructed VDM output. We evaluated NDF+ under reverberant conditions and compared it with representative conventional baselines. Results show that NDF+ consistently outperforms the baselines on both subtasks, while maintaining VDM reconstruction quality comparable to that of the original single-task NDF model. These findings indicate that NDF+ introduces an additional degree of freedom for diffuse sound control in the VDM reconstruction. In a stereo recording application, NDF+ provides controllable inter-channel level differences between left and right channels by adjusting the estimated diffuse component.
Primary: International Audio Laboratories Erlangen
All Institutions: International Audio Laboratories Erlangen, Fraunhofer IIS, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
The main contribution of this paper is the introduction of NDF+, a joint framework for neural directional filtering and diffuse sound extraction that enhances VDM reconstruction while allowing for effective control of diffuse sound components. This work represents a significant step forward in spatial audio processing, combining innovative methodologies with rigorous experimental validation to address key challenges in the field.
The paper introduces NDF+, a novel framework that combines neural directional filtering with diffuse sound extraction, effectively reformulating the VDM estimation into two coupled subtasks. The methodology employs a dual-mask architecture using LSTM networks to estimate coherent and diffuse components, which is a significant advancement over previous models that focused solely on VDM reconstruction. The approach is well-structured, with a clear explanation of the DNN architecture, training strategy, and loss functions, demonstrating a thoughtful integration of existing techniques with innovative modifications.
The experimental evaluation is comprehensive, comparing NDF+ against conventional baselines under various reverberant conditions. The results indicate that NDF+ consistently outperforms these baselines on both subtasks while maintaining VDM reconstruction quality. The use of objective metrics such as SDR and PESQ to measure performance adds rigor to the evaluation. However, the paper could benefit from more detailed qualitative assessments, such as user studies or subjective listening tests, to further validate the improvements in audio quality.
The paper provides a detailed description of the experimental setup, including the configurations of the microphone array, training data, and evaluation metrics. However, the absence of a public code repository or demo URL limits the reproducibility of the results. Including such resources would enhance the paper's impact and allow other researchers to validate and build upon the findings.
One limitation is the reliance on simulated environments for training and testing, which may not fully capture the complexities of real-world acoustic scenarios. Additionally, while the paper discusses the performance of NDF+ in stereo recording applications, it does not explore its scalability to larger microphone arrays or more complex sound environments.
The advancements presented in NDF+ have significant implications for spatial audio applications, particularly in enhancing the quality of recordings in reverberant environments. The ability to control diffuse sound components can improve immersive audio experiences in various fields, including virtual reality, telecommunications, and music production. The framework could also inspire further research into joint signal processing techniques in audio applications. The main contribution of this paper is the introduction of NDF+, a joint framework for neural directional filtering and diffuse sound extraction that enhances VDM reconstruction while allowing for effective control of diffuse sound components. This work represents a significant step forward in spatial audio processing, combining innovative methodologies with rigorous experimental validation to address key challenges in the field.
In audio generation evaluation, Fréchet Audio Distance (FAD) is a 2-Wasserstein distance with structural constraints for both primitives: the cost is a frozen embedding pullback whose invariance set hides severe artifacts, and the coupling is a Gaussian fit that dilutes rank-1 contamination relative to discrete OT. We propose Optimal Transport Audio Distance (OTAD), which corrects each primitive with one dedicated mechanism -- a residual Riemannian ground-metric adapter for the cost and entropic Sinkhorn optimal transport for the coupling. Across eight encoders under a four-axis protocol, coupling-only comparisons at $ε= 0.05$ show that Sinkhorn's rank-1 sensitivity exceeds FAD's by a factor of 1.9 to 3.6. Furthermore, OTAD achieves a higher mean Spearman correlation with audio-quality MOS (DCASE 2023 Task 7) than baseline metrics. As an intrinsic benefit of the discrete transport plan, OTAD yields per-sample diagnostics with AUROC $\ge 0.86$, a capability that scalar- or kernel-aggregated metrics structurally lack.
Primary: Sogang University
All Institutions: Sogang University
The paper presents a significant advancement in audio evaluation metrics by introducing OTAD, which effectively addresses the limitations of existing methods through innovative methodological contributions and rigorous empirical validation.
The proposed methodology introduces a novel Optimal Transport Audio Distance (OTAD) metric that addresses the limitations of existing metrics like Fréchet Audio Distance (FAD) by employing a dual correction mechanism: a learned Riemannian ground-metric adapter for the cost function and entropic Sinkhorn optimal transport for the coupling. This innovative approach allows for a more sensitive detection of artifacts in audio generation, which is crucial for applications in text-to-audio synthesis. The method is theoretically grounded and systematically validated through a comprehensive experimental design, including a factorial decomposition of the contributions from cost and coupling.
The experiments are robust, utilizing eight different encoders and a four-axis evaluation protocol to assess the performance of OTAD against FAD and KAD. The results indicate a significant improvement in sensitivity to rank-1 contamination and a higher correlation with human Mean Opinion Scores (MOS). The experiments also include per-sample diagnostics, which provide insights into the specific artifacts present in audio samples, highlighting the practical utility of OTAD in real-world applications.
The paper includes sufficient detail regarding the implementation of the OTAD metric and the experimental setup, including the datasets used (FSD50K and ESC-50) and the training of the adapters. The release of the OTAD toolkit on GitHub further enhances reproducibility, allowing other researchers to replicate the findings and utilize the metric in their own work.
The study acknowledges several limitations, including the reliance on a single listening test for MOS validation and the potential biases introduced by training on a specific dataset (FSD50K). Additionally, the performance of OTAD on music and speech domains remains untested, and the scalability of the method for larger datasets is not fully explored.
The introduction of OTAD has significant implications for the field of audio generation evaluation, providing a more nuanced and sensitive metric that can improve the quality of generated audio. This advancement could lead to better user experiences in applications such as music synthesis, sound design, and audio restoration. The methodology can also serve as a blueprint for future research in audio evaluation metrics across different domains. The paper presents a significant advancement in audio evaluation metrics by introducing OTAD, which effectively addresses the limitations of existing methods through innovative methodological contributions and rigorous empirical validation.
Symbolic music datasets with matched scores and performances are essential for many music information retrieval (MIR) tasks. Yet, existing resources often cover a narrow range of composers, lack performance variety, omit note-level alignments, or use inconsistent naming formats. This work presents PianoCoRe, a large-scale piano MIDI dataset that unifies and refines major open-source piano corpora. The dataset contains 250,046 performances of 5,625 pieces written by 483 composers, totaling 21,763 h of performed music. PianoCoRe is released in tiered subsets to support different applications: from large-scale analysis and pre-training (PianoCoRe-C and deduplicated PianoCoRe-B) to expressive performance modeling with note-level score alignment (PianoCoRe-A/A*). The note-aligned subset, PianoCoRe-A, provides the largest open-source collection of 157,207 performances aligned to 1,591 scores to date. In addition to the dataset, the contributions are: (1) a MIDI quality classifier for detecting corrupted and score-like transcriptions and (2) RAScoP, an alignment refinement pipeline that cleans temporal alignment errors and interpolates missing notes. The analysis shows that the refinement reduces temporal noise and eliminates tempo outliers. Moreover, an expressive performance rendering model trained on PianoCoRe demonstrates improved robustness to unseen pieces compared to models trained on raw or smaller datasets. PianoCoRe provides a ready-to-use foundation for the next generation of expressive piano performance research.
Primary: Skolkovo Institute of Science and Technology
All Institutions: Skolkovo Institute of Science and Technology
The main contribution of this paper is the introduction of the PianoCoRe dataset, a refined and comprehensive MIDI dataset that addresses the limitations of existing resources in symbolic music analysis. This work significantly enhances the foundation for future research in expressive piano performance modeling and music information retrieval, showcasing a meticulous approach to dataset curation and quality assessment.
The methodology presented in this paper is robust and comprehensive, detailing a systematic approach to curating and refining a large-scale piano MIDI dataset. The authors employ a multi-tiered strategy that includes deduplication, quality assessment, and note alignment refinement using the RAScoP pipeline. The integration of various existing datasets into a unified collection is particularly noteworthy, as it addresses the inconsistencies and limitations found in previous datasets. The use of a MIDI quality classifier to filter out corrupted transcriptions and the detailed description of the alignment process further enhance the methodological rigor.
The experiments conducted demonstrate the effectiveness of the proposed dataset and methodologies. The authors provide a thorough evaluation of the MIDI quality classifier, achieving a high macro F1 score, which indicates the classifier's reliability in distinguishing between performance qualities. Additionally, the application of the dataset in training an expressive performance rendering model shows significant improvements in robustness, suggesting that the dataset effectively supports advanced modeling tasks. However, specific quantitative results from the expressive performance rendering model could further strengthen the experimental validation.
The paper includes detailed descriptions of the dataset construction process, including data sources, matching methodologies, and quality assessment techniques. The authors provide a GitHub repository link for the project, which enhances reproducibility. However, the paper could benefit from including specific implementation details or code snippets to facilitate replication of the methodologies by other researchers.
One limitation identified is the reliance on existing datasets, which may still contain inherent biases or limitations that could affect the quality of the combined dataset. Additionally, while the RAScoP pipeline improves alignment, the paper does not fully address potential edge cases where alignment might still be problematic. The focus on public domain works may also limit the dataset's applicability to contemporary compositions.
The PianoCoRe dataset has the potential to significantly impact the field of music information retrieval and computational musicology by providing a comprehensive resource for training models in expressive performance rendering and analysis. Its tiered structure allows for diverse applications, from large-scale analysis to specific performance modeling tasks, thus fostering advancements in music generation and understanding. The main contribution of this paper is the introduction of the PianoCoRe dataset, a refined and comprehensive MIDI dataset that addresses the limitations of existing resources in symbolic music analysis. This work significantly enhances the foundation for future research in expressive piano performance modeling and music information retrieval, showcasing a meticulous approach to dataset curation and quality assessment.
We propose a plug-and-play framework for speech enhancement and separation that augments predictive methods with a generative speech prior. Our approach, termed Stochastic Interpolant Prior for Speech (SIPS), builds on stochastic interpolants and leverages their flexibility to bridge predictive and generative modeling. Specifically, we decompose the interpolation dynamics into a task-specific drift and a stochastic denoising component, allowing a predictive estimate to be integrated directly into the generative sampling process. This results in a mathematically grounded framework for combining strong pretrained predictors with the expressive power of generative models. To this end, we train a score model using only clean speech, yielding a degradation-agnostic prior that can be reused across tasks. During inference, the predictor provides a deterministic drift that steers the sampling process toward a task-consistent estimate, while the score model preserves perceptual naturalness. Unlike prior hybrid approaches, which typically rely on architecture-specific conditioning and are tied to particular predictors or degradation settings, SIPS provides a unified framework that generalizes across predictors and additive degradation tasks. We demonstrate its effectiveness for both speech enhancement and speech separation using recent predictors such as SEMamba and FlexIO. The proposed method consistently improves perceptual quality, achieving gains up +1.0 NISQA for speech separation.
Primary: MERL
All Institutions: MERL
The paper presents a novel framework that bridges predictive and generative modeling for speech enhancement and separation, demonstrating significant improvements in perceptual quality while maintaining competitive performance on traditional metrics. The comprehensive methodology and robust experimental validation position this work as a meaningful contribution to the field of machine learning in audio processing.
The proposed Stochastic Interpolant Prior for Speech (SIPS) framework effectively integrates predictive and generative modeling approaches, addressing the limitations of both paradigms by introducing a mathematically grounded decomposition of interpolation dynamics. This innovative methodology allows for a flexible and efficient plug-and-play integration with existing predictors, enhancing the perceptual quality of speech enhancement and separation tasks while maintaining fidelity to the original signals.
The experiments conducted demonstrate the efficacy of SIPS across various tasks, including speech enhancement and separation, using multiple state-of-the-art predictors. The results indicate consistent improvements in non-intrusive perceptual quality metrics, alongside competitive performance in reference-based metrics, showcasing the robustness and versatility of the proposed method.
The paper provides a clear implementation of the proposed method, including detailed descriptions of the experimental setup, data representation, and training procedures. The availability of the implementation on GitHub enhances reproducibility, allowing other researchers to validate and build upon the findings.
One limitation is the reliance on clean speech data for training the generative prior, which may affect performance in real-world scenarios with diverse degradation types. Additionally, while the method shows promise, further exploration of its generalization capabilities across different audio domains is warranted.
The SIPS framework has significant implications for various applications in speech processing, including telecommunications, assistive technologies, and audio content creation. By improving speech quality in challenging conditions, this work can enhance user experiences in voice communication systems and contribute to advancements in automatic speech recognition and natural language processing. The paper presents a novel framework that bridges predictive and generative modeling for speech enhancement and separation, demonstrating significant improvements in perceptual quality while maintaining competitive performance on traditional metrics. The comprehensive methodology and robust experimental validation position this work as a meaningful contribution to the field of machine learning in audio processing.
Large audio language models (LALMs) are increasingly used to reason over long audio clips, yet deployment often compresses audio before inference to reduce memory and latency. The risk is that compression can leave aggregate accuracy acceptable while sharply degrading answers for a deployment-critical query family. We study answer-preserving audio compression, judging a compressor by the excess answer-error it induces, especially for the worst-affected family. We formulate this theoretically as a compressor acceptance-rejection criterion, derive a practical sign-off protocol that returns compression budgets satisfying worst-family checks with statistical confidence, and evaluate it on five multiple-choice audio question-answering benchmarks with two Qwen-based backbones. The protocol exposes hidden family-level damage, shows that the chosen query-family partition can change the approved budget, and identifies regimes where query-conditioned compression helps maintain answer preservation.
Primary: Technion--Israel Institute of Technology
All Institutions: Technion--Israel Institute of Technology
The main contribution of this paper is the introduction of a framework for task-aware answer-preserving audio compression, which addresses the critical challenge of maintaining answer quality in large audio language models under compression constraints. This work significantly advances the understanding of audio compression impacts on model performance and provides a practical methodology for evaluating and ensuring answer preservation across diverse query families.
The methodology presented in this paper is robust and well-structured. The authors introduce a theoretical framework for task-aware answer-preserving audio compression, which is a novel approach to evaluating audio compression techniques in the context of large audio language models (LALMs). The paper formulates a compressor acceptance-rejection criterion and derives a practical sign-off protocol that incorporates statistical confidence, which is a significant contribution to the field. The approach is grounded in a solid theoretical foundation, linking practical deployment configurations to answer preservation metrics. The use of paired evaluations and the focus on worst-family checks are particularly noteworthy, as they address the critical issue of performance degradation across different query families.
The experimental evaluation is comprehensive, utilizing five multiple-choice audio question-answering benchmarks. The authors effectively demonstrate the applicability of their framework and the importance of considering family-level performance rather than relying solely on average metrics. The results reveal significant insights into how compression can affect different query families, showcasing the hidden damage that can occur when using average performance metrics. The experiments are well-designed, and the analysis is thorough, providing empirical support for the theoretical claims made in the paper.
The paper provides detailed descriptions of the experimental setup, including the datasets, models, and evaluation metrics used. However, there are some limitations regarding the availability of code and data, as no URLs for project repositories or demo pages are provided. This lack of resources may hinder reproducibility for other researchers looking to validate or build upon the findings.
The paper acknowledges several limitations, including the potential for query-family coarsening and the challenges of estimating true Bayes risks due to calibration errors and prompt sensitivity. Additionally, the framework's applicability to different languages, longer audio clips, or varying deployment scenarios is not fully established, which may limit its generalizability.
The proposed framework has significant implications for the deployment of audio language models in real-world applications, particularly in scenarios where audio compression is necessary for efficiency. By emphasizing the importance of answer preservation across different query families, this work could influence future research and development in audio processing, machine learning, and multimodal systems. The findings could lead to improved audio compression techniques that better maintain the integrity of information critical for specific tasks. The main contribution of this paper is the introduction of a framework for task-aware answer-preserving audio compression, which addresses the critical challenge of maintaining answer quality in large audio language models under compression constraints. This work significantly advances the understanding of audio compression impacts on model performance and provides a practical methodology for evaluating and ensuring answer preservation across diverse query families.
Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Shanghai Innovation Institute, Tencent, Independent Researcher, Peking University, Tianjin University, Zhejiang University
WavCube presents a novel approach to unify speech understanding and generation through a compact continuous latent representation. This paper makes a substantial contribution to the field by addressing the compatibility challenges between semantic and acoustic features, demonstrating its effectiveness through rigorous experimentation across multiple benchmarks.
The methodology employed in WavCube is innovative, utilizing a two-stage training scheme that effectively addresses the challenges of integrating semantic and acoustic representations. The first stage compresses high-dimensional SSL features into a compact latent space, while the second stage enriches this latent space with fine-grained acoustic details. This approach is well-justified and systematically tackles the inherent flaws of existing SSL representations, making it a significant contribution to the field.
The experiments conducted are comprehensive and well-structured, demonstrating WavCube's performance across various tasks, including speech understanding, reconstruction, and generation. The results show that WavCube achieves competitive performance against existing methods, indicating its effectiveness and robustness. The use of benchmarks like SUPERB and the detailed evaluation metrics further enhance the credibility of the findings.
The paper provides sufficient details regarding the methodology and experimental setup, including the datasets and training configurations used. However, the lack of a demo or interactive component may hinder some aspects of reproducibility for practitioners who wish to implement the model.
While the paper presents a strong framework, it does not explicitly discuss potential limitations or assumptions underlying the proposed approach. For instance, the performance drop due to dimensionality reduction and the reliance on specific datasets could be areas of concern that warrant further exploration.
The implications of WavCube are significant, as it offers a unified framework for speech processing that could enhance applications in voice synthesis, speech recognition, and multimodal interactions. By bridging the gap between understanding and generation, WavCube could pave the way for more integrated and efficient speech technologies. WavCube presents a novel approach to unify speech understanding and generation through a compact continuous latent representation. This paper makes a substantial contribution to the field by addressing the compatibility challenges between semantic and acoustic features, demonstrating its effectiveness through rigorous experimentation across multiple benchmarks.
In this paper, we present X-Voice, a 0.4B multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages. X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation. To eliminate the reliance on prompt text without complex preprocessing like forced alignment, we design a two-stage training paradigm. In Stage 1, we establish X-Voice$_{\text{s1}}$ through standard conditional flow-matching training and use it to synthesize 10K hours of speaker-consistent segments as audio prompts. In Stage 2, we fine-tune on these audio pairs with prompt text masked to derive X-Voice$_{\text{s2}}$, which enables zero-shot voice cloning without requiring transcripts of audio prompts. Architecturally, we extend F5-TTS by implementing a dual-level injection of language identifiers and decoupling and scheduling of Classifier-Free Guidance to facilitate multilingual speech synthesis. Subjective and objective evaluation results demonstrate that X-Voice outperforms existing flow-matching based multilingual systems like LEMAS-TTS and achieves zero-shot cross-lingual cloning capabilities comparable to billion-scale models such as Qwen3-TTS. To facilitate research transparency and community advancement, we open-source all related resources.
Primary: Zhejiang University
All Institutions: Zhejiang University, Beijing Haitian Ruisheng Science Technology Ltd, Center for Language and Speech Processing, Fudan University, Geely Automobile Research Institute (Ningbo) Company Ltd, MoE Key Lab of Artificial Intelligence, Shanghai Innovation Institute, Shanghai Jiao Tong University, X-LANCE Lab
The paper presents X-Voice, a novel multilingual zero-shot voice cloning model that significantly advances the capabilities of TTS systems across 30 languages. The methodology, which includes a two-stage training process and innovative architectural enhancements, addresses critical limitations in existing systems, making it a valuable contribution to the field of machine learning and audio processing.
The paper introduces a two-stage training paradigm for zero-shot voice cloning, which is a significant advancement in the field. The first stage focuses on building a robust multilingual backbone using a large corpus, while the second stage fine-tunes the model using synthetic audio prompts without the need for reference transcripts. This approach effectively addresses the challenges of multilingual TTS systems, particularly the reliance on aligned text and audio, which is often problematic for low-resource languages. The introduction of dual-level language injection and decoupled classifier-free guidance further enhances the model's ability to maintain speaker identity and prosodic accuracy across languages.
The experimental results are comprehensive, comparing X-Voice against several state-of-the-art models across multiple languages. The use of both subjective and objective evaluation metrics, including WER, SIM-o, IMOS, and SMOS, provides a well-rounded assessment of the model's performance. The results indicate that X-Voice achieves competitive performance, particularly in low-resource languages, while also demonstrating improvements in intelligibility and speaker consistency compared to existing systems. The release of a new evaluation benchmark with human annotations adds significant value to the research community.
The paper provides detailed implementation details, including model configurations, training setups, and evaluation protocols, which enhances reproducibility. The authors have also open-sourced their training corpus and evaluation benchmarks, fostering transparency and allowing other researchers to build upon their work.
Despite its strengths, the model still faces challenges in preserving speaker similarity in certain phonological contexts, indicating a trade-off between accent suppression and timbre preservation. Additionally, the handling of intra-sentential code-switching is noted as an area for future improvement. The reliance on high-quality synthetic data in the fine-tuning stage may also limit the model's applicability in scenarios where such data is not available.
The advancements presented in this paper have the potential to democratize high-fidelity TTS technology, making it accessible for a wider range of languages, including low-resource ones. The implications extend to various applications, such as personalized voice assistants, language learning tools, and accessibility technologies for individuals with speech impairments. The open-sourcing of resources could significantly accelerate research in multilingual TTS systems and contribute to the development of more inclusive technologies. The paper presents X-Voice, a novel multilingual zero-shot voice cloning model that significantly advances the capabilities of TTS systems across 30 languages. The methodology, which includes a two-stage training process and innovative architectural enhancements, addresses critical limitations in existing systems, making it a valuable contribution to the field of machine learning and audio processing.
Quantum machine learning has emerged as a promising tool for pattern recognition, yet many audio-focused approaches still treat spectrograms as generic images and do not explicitly exploit their time-frequency structure. We propose Q-Patch, a quantum feature map tailored to audio that encodes local time-frequency patches from mel-spectrograms into quantum states using shallow, hardware-efficient circuits with adjacency-aware entanglement. Each selected patch is summarized by a compact four-dimensional acoustic descriptor and mapped to a four-qubit circuit with depth at most three, enabling practical quantum kernel construction under near-term constraints. We evaluate Q-Patch on an audio spoofing detection task using a controlled, balanced protocol and compare it with size-matched classical baselines. Q-Patch improves discrimination between bona fide and spoofed samples, achieving an area under the receiver operating characteristic curve (AUROC) of 0.87, compared with 0.82 for a radial basis function support vector machine (RBF-SVM) trained on the same patch-level features. Kernel-space analysis further reveals a clear class structure, with cross-class similarity around 0.615 and within-class self-similarity of 1.00. Overall, Q-Patch provides a practical framework for incorporating time-frequency-aware representations into quantum kernel learning for audio authenticity assessment in low-resource settings.
Primary: Potomac Quantum
All Institutions: Potomac Quantum, United International University, University of Maryland, Monash University, University of the Sunshine Coast
The paper presents Q-Patch, a quantum feature-mapping framework for audio spoofing detection that effectively utilizes time-frequency structures in spectrograms. This innovative approach, combined with rigorous experimental validation, positions the work as a meaningful contribution to the field of audio deepfake detection and quantum machine learning.
The methodology introduces Q-Patch, a novel quantum feature mapping framework specifically designed for audio spoofing detection. It effectively utilizes local time-frequency patches from mel-spectrograms, which is a significant improvement over treating spectrograms as generic images. The use of shallow, hardware-efficient quantum circuits with adjacency-aware entanglement is innovative, as it addresses practical constraints of near-term quantum computing. The approach to summarize patches into compact four-dimensional descriptors before quantum embedding is well thought out, allowing for efficient processing while maintaining relevant information.
The experimental evaluation is conducted on a balanced dataset derived from LJ Speech, which includes both bona fide and spoofed audio samples. The results indicate that Q-Patch outperforms classical baselines, achieving an AUROC of 0.87 compared to 0.82 for RBF-SVM. The analysis of kernel-space structure further supports the effectiveness of the proposed method, showing clear class separability. However, the limited dataset size (100 samples) raises concerns about the generalizability of the results, which should be addressed in future work.
The paper provides a detailed description of the methodology, including data preparation, feature extraction, and quantum embedding processes. However, the absence of code or a project URL limits the reproducibility of the results. Future work should include sharing the implementation details or code to enable other researchers to replicate the findings.
The study's limitations include the small dataset size, which may not capture the full diversity of real-world audio spoofing attacks. The controlled nature of the spoof generation (using additive noise and spectral distortions) may not reflect the complexities of actual spoofing methods. Additionally, the results are based on ideal quantum simulations, which may not translate directly to performance on physical quantum hardware.
The proposed Q-Patch framework has the potential to significantly impact the field of audio deepfake detection by introducing quantum machine learning techniques that leverage time-frequency structures. This could lead to more robust detection methods that are particularly useful in low-resource settings. As quantum computing technology advances, the framework may become increasingly applicable in real-world scenarios, enhancing security against audio spoofing. The paper presents Q-Patch, a quantum feature-mapping framework for audio spoofing detection that effectively utilizes time-frequency structures in spectrograms. This innovative approach, combined with rigorous experimental validation, positions the work as a meaningful contribution to the field of audio deepfake detection and quantum machine learning.
The Massive Sound Embedding Benchmark (MSEB) has emerged as a standard for evaluating the functional breadth of audio models. While initial baselines focused on specialized encoders, the shift toward "audio-native" Large Language Models (LLMs) suggests a new paradigm where a single multimodal backbone may replace complex, task-specific pipelines. This paper provides a rigorous empirical evaluation of leading LLMs - including members from the Gemini and GPT families - across the eight core MSEB capabilities to assess their efficacy and audio-text parity. Our results indicate that while a significant modality gap persists regarding performance and robustness, the empirical evidence for an "optimal" modeling approach remains inconclusive. Ultimately, the choice between audionative and cascaded architectures depends heavily on specific use-case requirements and the underlying assumptions regarding latency, cost, and reasoning depth.
Primary: Google
All Institutions: Google USA & Germany
The main contribution of this paper is the rigorous empirical evaluation of leading audio-native LLMs on the MSEB, providing valuable insights into their performance and the challenges of achieving audio-text parity. The comprehensive analysis of methodologies and results positions this work as a significant step forward in the integration of audio processing within the framework of large language models, addressing both theoretical and practical aspects of the field.
The paper presents a comprehensive methodology for applying large language models (LLMs) to the Massive Sound Embedding Benchmark (MSEB), detailing a systematic approach to task-specific prompting and evaluation across diverse audio tasks. The methodology is well-structured, with clear definitions of tasks, input/output formats, and considerations for model performance. The iterative refinement of prompt templates through interactions with models like Gemini 3 demonstrates a thoughtful approach to optimizing LLMs for audio tasks. However, the paper could benefit from a more detailed discussion of the limitations of the chosen methodologies, particularly regarding the adaptability of LLMs to non-generative tasks.
The experimental evaluation is robust, covering a wide range of models and tasks, with detailed performance metrics provided for each evaluation. The use of a diverse set of datasets, including multilingual and varied acoustic environments, enhances the reliability of the results. The paper effectively compares audio-native LLMs with traditional cascaded systems, providing insights into their relative strengths and weaknesses. However, the analysis of results could be improved by including more visual aids (e.g., graphs) to illustrate performance trends across tasks and models.
The paper mentions the open-source nature of the MSEB toolkit and provides a link to the GitHub repository, which is a positive aspect for reproducibility. However, the paper lacks detailed implementation specifics, such as hyperparameter settings, training protocols, and the exact versions of models used, which could hinder full reproducibility for other researchers.
The paper acknowledges the significant modality gap that persists between audio and text processing, which is a critical limitation. Additionally, the authors note the challenges in achieving consistent performance across different locales and acoustic conditions, indicating that the models may not generalize well in real-world applications. The potential for test data contamination is also a significant concern that could skew results.
The findings of this research have significant implications for the development of audio processing systems, particularly in enhancing the capabilities of LLMs in understanding and reasoning about audio data. The establishment of the MSEB as a benchmark could drive further research and innovation in the field, promoting the development of more robust and versatile audio-native models. The open-source nature of the toolkit encourages community engagement and collaboration, which could accelerate advancements in auditory intelligence. The main contribution of this paper is the rigorous empirical evaluation of leading audio-native LLMs on the MSEB, providing valuable insights into their performance and the challenges of achieving audio-text parity. The comprehensive analysis of methodologies and results positions this work as a significant step forward in the integration of audio processing within the framework of large language models, addressing both theoretical and practical aspects of the field.
This study presents a bio inspired signal processing framework for robust Underwater Acoustic Target Recognition (UATR). The latest state of the art methods often fail to resolve dense low frequency harmonic structures in vessel propulsion signals under high noise conditions, which is addressed by the proposed framework using a biologically inspired Gammatone filter bank that emulates the cochlea nonlinear frequency selectivity. By distributing filters according to the Equivalent Rectangular Bandwidth (ERB) scale, the framework achieves a high fidelity representation of engine radiated tonals while effectively suppressing isotropic ambient interference. The resulting Cochleagram features are processed by a lightweight, custom designed Convolutional Neural Network (CNN) that leverages large receptive fields to integrate spectral-temporal continuities. Experimental results on the VTUAD dataset demonstrate a state of the art classification accuracy of 98.41%, outperforming Continuous Wavelet Transform and Mel Frequency Cepstral Coefficients baselines by 3.5% and 7.7% respectively. Furthermore, the framework achieves an inference latency of only 0.77 ms and a 0.971 Cohen Kappa score, validating its efficacy for real time deployment on autonomous, low-power sonar hardware.
Primary: Centre for Applied Research in Electronics (CARE)
All Institutions: Central Research Laboratory, Bharat Electronics Limited, Ghaziabad, India, Centre for Applied Research in Electronics (CARE), IIT Delhi, India
The main contribution of this paper is the development of a bio-inspired Gammatone-CNN framework for underwater acoustic target classification, achieving state-of-the-art performance through innovative feature extraction techniques. This research significantly advances the field of underwater acoustics by providing a method that combines biological principles with modern machine learning, demonstrating the potential for improved classification accuracy in challenging acoustic environments.
The paper introduces a novel bio-inspired Gammatone filter bank for underwater acoustic target classification, leveraging the non-linear frequency selectivity of the cochlea. The methodology prioritizes feature extraction over architectural complexity, employing a lightweight CNN that effectively integrates spectral-temporal features. The use of the Equivalent Rectangular Bandwidth (ERB) scale for filter distribution is particularly innovative, allowing for high fidelity in low-frequency representation, which is crucial for underwater acoustics. The mathematical foundations of the Gammatone filter and the detailed description of the Cochleagram formation process provide a solid basis for the proposed approach.
The experimental validation is robust, utilizing the VTUAD dataset to demonstrate the framework's effectiveness. Achieving a classification accuracy of 98.41% and a Cohen Kappa score of 0.971 indicates strong performance and reliability. The comparative analysis against established methods like Continuous Wavelet Transform and Mel Frequency Cepstral Coefficients shows significant improvements, reinforcing the proposed method's superiority. The inclusion of diverse metrics such as ROC curves and confusion matrices adds depth to the evaluation.
The paper provides detailed information on the experimental setup, including dataset partitioning, feature extraction parameters, and model architecture. However, the absence of a publicly available code repository limits reproducibility. Future work should consider sharing implementation details to facilitate validation by other researchers.
While the proposed framework shows remarkable performance, it may struggle with class imbalance, particularly for underrepresented classes like Passengership. The reliance on a specific dataset (VTUAD) may also limit generalizability to other underwater environments. Additionally, the computational efficiency on standard CPUs, while acceptable, could be a concern for real-time applications in more constrained environments.
The implications of this research extend to maritime security, ecological monitoring, and autonomous underwater vehicles (AUVs). By improving underwater target recognition, the framework can enhance surveillance capabilities and contribute to the protection of marine ecosystems. The low-power, real-time processing capabilities make it suitable for deployment in resource-constrained environments. The main contribution of this paper is the development of a bio-inspired Gammatone-CNN framework for underwater acoustic target classification, achieving state-of-the-art performance through innovative feature extraction techniques. This research significantly advances the field of underwater acoustics by providing a method that combines biological principles with modern machine learning, demonstrating the potential for improved classification accuracy in challenging acoustic environments.
The rapid advancement of generative audio models has outpaced the development of robust evaluation methodologies. Existing objective metrics and general multimodal large language models (MLLMs) often struggle with domain generalization, zero-shot capabilities, and instructional flexibility. To address these bottlenecks, we propose JASTIN, a generalizable, instruction-driven audio evaluation framework that formulates audio assessment as a self-instructed reasoning task. JASTIN bridges a frozen high-performance audio encoder with a fine-tuned LLM backbone via a trainable audio adapter. To ensure robust zero-shot generalization, we introduce a comprehensive instruction following data preparation pipeline, incorporating Multi-Source, Multi-Task, Multi-Calibration, and Multi-Description data. Experimental results demonstrate that JASTIN achieves state-of-the-art Pearson and Spearman correlations with human subjective ratings. It consistently outperforms general MLLMs across speech, sound, music, and out-of-domain evaluation tasks without the need for task-specific retraining.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, MoE Key Laboratory of Artificial Intelligence, AI Institute
The main contribution of this paper is the introduction of JASTIN, a novel instruction-driven framework for zero-shot audio evaluation that significantly enhances the evaluation process by integrating multimodal LLMs with advanced audio processing techniques. This work represents a meaningful advancement in the field of audio evaluation, addressing critical challenges and setting a new standard for future research.
The proposed JASTIN framework innovatively integrates a frozen high-performance audio encoder with a fine-tuned LLM backbone through a trainable audio adapter, addressing the limitations of existing evaluation metrics by employing a self-instructed reasoning paradigm. The comprehensive data preparation pipeline, which includes multi-source, multi-task, multi-calibration, and multi-description strategies, enhances the model's zero-shot generalization capabilities, making it adaptable to various audio evaluation tasks without the need for task-specific retraining.
The experimental results demonstrate that JASTIN achieves state-of-the-art Pearson and Spearman correlations with human subjective ratings across diverse audio domains, including speech, sound, and music. The framework consistently outperforms both traditional metrics and general MLLMs, showcasing its robustness and effectiveness in real-world applications. The evaluation on out-of-domain tasks further emphasizes its generalization capabilities, which is a significant advancement in the field.
The authors have provided detailed implementation information, including training configurations, data preparation methods, and evaluation metrics, which enhances the reproducibility of their results. However, the lack of a demo URL limits immediate accessibility for other researchers to test the framework.
While the paper presents a comprehensive framework, it does not address potential biases in the training data or the limitations of the LLMs used. Additionally, the model's performance on highly specialized audio tasks may still require further validation.
The JASTIN framework has the potential to revolutionize audio evaluation methodologies by providing a more flexible and generalizable approach. Its implications extend to various applications in audio synthesis, music generation, and speech processing, enabling more efficient and scalable evaluation processes in these domains. The main contribution of this paper is the introduction of JASTIN, a novel instruction-driven framework for zero-shot audio evaluation that significantly enhances the evaluation process by integrating multimodal LLMs with advanced audio processing techniques. This work represents a meaningful advancement in the field of audio evaluation, addressing critical challenges and setting a new standard for future research.
While the spatial directivity of multichannel speech enhancement algorithms improves with the number of microphones, fitting large capture arrays into real-world edge devices is typically limited by physical constraints. To overcome this limitation, we propose Spatial-Magnifier, a neural network designed to generate virtual microphone (VM) signals from a limited set of real microphone (RM) measurements. Moreover, we introduce the Spatial Audio Representation Learning (SARL) framework, which leverages estimated VM signals and features to condition a downstream speech enhancement system. Experimental results demonstrate that the proposed framework outperforms existing spatial upsampling baselines across various speech extraction systems, including end-to-end multichannel speech enhancement and neural beamforming. The proposed method nearly recovers the oracle performance achieved when all microphones are available.
Primary: Korea Advanced Institute of Science and Technology (KAIST)
All Institutions: Korea Advanced Institute of Science and Technology (KAIST), Meta Reality Labs Research
The paper introduces Spatial-Magnifier, a neural network for spatial upsampling in multichannel speech enhancement, and the SARL framework, significantly enhancing downstream speech processing tasks. The innovative approach and rigorous experimental validation position this work as a valuable contribution to the field of audio signal processing and machine learning.
The paper presents a novel neural network architecture, Spatial-Magnifier, which effectively generates virtual microphone signals from real microphone measurements. It introduces the Spatial Audio Representation Learning (SARL) framework, which enhances the conditioning of downstream speech enhancement tasks by leveraging both estimated virtual microphone signals and features. The use of a GAN-based approach and the incorporation of selection and dynamic channel allocation modules are innovative aspects that contribute to the flexibility and efficiency of the model. The methodology is well-structured, with clear definitions and a logical flow from problem identification to proposed solutions.
The experiments are comprehensive, utilizing a well-defined dataset and a robust experimental setup to evaluate the performance of the proposed methods. The authors conduct ablation studies and comparisons with existing baselines, demonstrating the effectiveness of their approach across different configurations and tasks. The results indicate significant improvements in performance metrics such as SI-SDR, SNR, PESQ, and STOI, showcasing the technical superiority of the proposed methods over traditional approaches.
The paper provides sufficient details regarding the experimental setup, including the architecture parameters, training procedures, and evaluation metrics. However, the absence of a publicly accessible code repository or demo URL limits the reproducibility of the results. Future work could benefit from sharing the implementation to facilitate validation by the research community.
One limitation is the reliance on simulated data for training and evaluation, which may not fully capture the complexities of real-world environments. Additionally, while the proposed methods show promise, the performance in highly dynamic or noisy environments remains to be thoroughly evaluated. The computational efficiency, while improved, could still be a concern for deployment on resource-constrained devices.
The proposed methods have significant implications for real-world applications in speech enhancement, particularly in consumer electronics such as AR glasses and hearing aids. By enabling effective multichannel speech enhancement with fewer microphones, the work addresses a critical need for improved audio capture in compact devices. The advancements in spatial audio processing could also benefit various fields, including telecommunications, virtual reality, and assistive technologies. The paper introduces Spatial-Magnifier, a neural network for spatial upsampling in multichannel speech enhancement, and the SARL framework, significantly enhancing downstream speech processing tasks. The innovative approach and rigorous experimental validation position this work as a valuable contribution to the field of audio signal processing and machine learning.
Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict. Crucially, conflicts differ in resolvability: benign conflicts stem from missing, weak, or ambiguous cues and can be mitigated by cross-modal calibration, while severe conflicts arise from intrinsically contradictory (e.g., sarcasm) or misleading signals, for which forced fusion may amplify errors. Recognizing this, we propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities. Path I (Affective Fusion Distiller, AFD) performs reverse distillation from audio/visual teachers to a textual student using temporally weighted class evidence, thereby enhancing representation-level calibration and improving fusion when alignment is beneficial. Path II (Affective Discernment Agent, ADA) formulates MER as a contextual bandit that selects among fusion and unimodal predictions based on a dual-view state and a calibration-aware reward, enabling decision-level arbitration under irreconcilable conflicts without requiring per-modality reliability labels. By taking into account the full multimodal context and coupling soft calibration with hard arbitration, DCR reconciles conflicts that can be aligned while bypassing misleading modalities when fusion is harmful. Across five benchmarks covering both dialogue-level and clip-level MER, DCR consistently outperforms competitive baselines or achieves highly competitive results. Further ablations, conflict-specific subset evaluation, and modality-selection analysis verify that AFD and ADA are complementary and jointly improve robust conflict-aware emotion recognition.
Primary: Hefei University of Technology
All Institutions: Hefei University of Technology, Singapore Management University, Nanyang Technological University, MIT Media Lab
The main contribution of this paper is the introduction of the Dual-Path Conflict Resolution framework, which innovatively addresses modality conflicts in multimodal emotion recognition by employing a dual-path approach that distinguishes between benign and severe conflicts. This comprehensive analysis highlights the technical contributions, methodological rigor, and potential impact of the research on the field of affective computing.
The proposed Dual-Path Conflict Resolution (DCR) framework is a significant advancement in multimodal emotion recognition (MER). It effectively distinguishes between benign and severe modality conflicts, employing two distinct paths (AFD and ADA) to handle these conflicts appropriately. AFD utilizes knowledge distillation to enhance textual representations with non-verbal cues, while ADA employs a contextual bandit approach for decision-level arbitration. This dual-path strategy is innovative as it shifts the focus from traditional fusion methods that may amplify errors to a more nuanced conflict-aware approach. The methodology is well-structured, with clear definitions of conflict types and a comprehensive explanation of how each path operates.
The experimental evaluation is robust, covering five diverse benchmarks that include both dialogue-level and clip-level datasets. The results consistently demonstrate that DCR outperforms competitive baselines, indicating its effectiveness across different contexts. The paper includes detailed ablation studies that validate the contributions of each component within the DCR framework, further strengthening the findings. The use of multiple evaluation metrics enhances the reliability of the results.
The paper provides sufficient implementation details, including the architecture, training protocols, and datasets used. However, the absence of a public demo or detailed code repository at the time of review limits reproducibility. The authors mention that the source code and models will be released, which is a positive step towards enhancing reproducibility.
One limitation of the study is the reliance on heuristic approximations for defining conflict severity, which may not capture the full complexity of modality interactions in real-world scenarios. Additionally, while the framework shows strong performance, its effectiveness in highly nuanced or ambiguous emotional contexts remains to be fully explored.
The DCR framework has significant implications for various applications, including human-computer interaction, healthcare, and robotics, where accurate emotion recognition is crucial. By addressing modality conflicts more effectively, this work could lead to more reliable affective computing systems that better understand human emotions. The main contribution of this paper is the introduction of the Dual-Path Conflict Resolution framework, which innovatively addresses modality conflicts in multimodal emotion recognition by employing a dual-path approach that distinguishes between benign and severe conflicts. This comprehensive analysis highlights the technical contributions, methodological rigor, and potential impact of the research on the field of affective computing.
Recent progress in diffusion-based audio generation and restoration has substantially improved performance across heterogeneous conditioning regimes, including text-conditioned audio generation and audio-conditioned super-resolution. However, training audio diffusion models remains computationally expensive, and most existing pipelines still rely on static optimization recipes that treat the relative importance of training signals as fixed throughout learning. In this work, we argue that a major source of inefficiency lies in the evolving balance between semantic acquisition and generation-oriented refinement. Early training places stronger emphasis on acquiring condition-aligned semantic structure and coarse global organization, whereas later training increasingly emphasizes temporal consistency, perceptual fidelity, and fine-detail refinement. To characterize this evolving balance, we introduce a progress-based regime variable derived from the training-time slope of an SSL-space discrepancy, which measures semantic progress during training. Based on this signal, we develop three complementary stage-aware mechanisms: decayed SSL guidance for early semantic bootstrapping, self-adaptive timestep sampling driven by the regime variable, and structure-aware regularization activated from convergent grouped organization in parameter space. We evaluate these mechanisms on text-conditioned audio generation and audio-conditioned super-resolution. Across both settings, the proposed stage-aware strategies improve convergence behavior and yield gains on the primary generation and spectral reconstruction metrics over standard static baselines. These results support the view that efficient audio diffusion training can benefit from treating external guidance, internal organization, and optimization emphasis as stage-dependent components rather than fixed ingredients.
Primary: China Pharmaceutical University
All Institutions: China Pharmaceutical University, University of Science and Technology of China
The paper presents a novel stage-adaptive framework for audio diffusion modeling, significantly enhancing training efficiency and model performance. The comprehensive methodology and experimental validation contribute valuable insights to the field, although concerns regarding reproducibility and the need for broader applicability remain.
The paper introduces a stage-aware perspective on audio diffusion training, which is a significant methodological innovation. The authors propose three complementary mechanisms—decayed SSL guidance, self-adaptive timestep sampling, and structure-aware regularization—each designed to adapt the training process based on the evolving needs of the model. This approach is well-justified and supported by a clear theoretical framework, utilizing a regime variable to monitor semantic progress. The proposed methods are distinct from traditional static optimization techniques, marking a notable advancement in the field of audio diffusion modeling.
The experiments are comprehensive, evaluating the proposed methods on both text-conditioned audio generation and audio-conditioned super-resolution. The use of multiple metrics (e.g., FAD, KL divergence, and spectral reconstruction metrics) provides a robust assessment of performance improvements. The results consistently demonstrate that the stage-aware mechanisms outperform static baselines, highlighting their effectiveness. However, the paper could benefit from additional experiments to further validate the findings across diverse datasets and conditions.
The paper lacks explicit details regarding the implementation and availability of code or datasets, which raises concerns about reproducibility. While the methodology is well-documented, the absence of a project URL or demo limits the ability of other researchers to replicate the results or build upon the work.
One limitation is the reliance on a single frozen SSL encoder, which may restrict the generalizability of the findings. Additionally, while the results show improvements in convergence and quality metrics, the paper does not sufficiently address the computational overhead introduced by the proposed mechanisms. The authors also acknowledge that gains in certain metrics (e.g., SISNR) were less pronounced, suggesting that the approach may not uniformly enhance all aspects of audio quality.
The findings have significant implications for the development of efficient audio generation systems, particularly in applications requiring high-quality audio synthesis and restoration. By demonstrating that training efficiency can be improved through a stage-aware approach, this work may influence future research directions in generative modeling and audio processing. The insights gained could also be applicable to other domains where dynamic adaptation of training strategies is beneficial. The paper presents a novel stage-adaptive framework for audio diffusion modeling, significantly enhancing training efficiency and model performance. The comprehensive methodology and experimental validation contribute valuable insights to the field, although concerns regarding reproducibility and the need for broader applicability remain.
High-quality singing annotations are fundamental to modern Singing Voice Synthesis (SVS) systems. However, obtaining these annotations at scale through manual labeling is unrealistic due to the substantial labor and musical expertise required, making automatic annotation highly necessary. Despite their utility, current automatic transcription systems face significant challenges: they often rely on complex multi-stage pipelines, struggle to recover text-note alignments, and exhibit poor generalization to out-of-distribution (OOD) singing data. To alleviate these issues, we present VocalParse, a unified singing voice transcription (SVT) model built upon a Large Audio Language Model (LALM). Specifically, our novel contribution is to introduce an interleaved prompting formulation that jointly models lyrics, melody, and word-note correspondence, yielding a generated sequence that directly maps to a structured musical score. Furthermore, we propose a Chain-of-Thought (CoT) style prompting strategy, which decodes lyrics first as a semantic scaffold, significantly mitigating the context disruption problem while preserving the structural benefits of interleaved generation. Experiments demonstrate that VocalParse achieves state-of-the-art SVT performance on multiple singing datasets. The source code and checkpoint are available at https://github.com/pymaster17/VocalParse.
Primary: Xi'an Jiaotong University
All Institutions: Xi'an Jiaotong University, Nanyang Technological University, Tianjin University, Ant Group, Zhejiang University
The main contribution of this paper is the development of VocalParse, a unified and scalable singing voice transcription framework that effectively integrates lyrics and melody transcription using advanced prompting strategies and a novel data collection pipeline. This work represents a significant step forward in addressing the challenges of automatic singing voice transcription, with implications for both academic research and practical applications in music technology.
The paper introduces VocalParse, a unified singing voice transcription model leveraging a Large Audio Language Model (LALM). The methodology is innovative, particularly with the interleaved prompting formulation that integrates lyrics and melody in a structured manner, addressing the challenges of traditional multi-stage pipelines. The Chain-of-Thought (CoT) prompting strategy is a significant advancement, allowing for better semantic continuity in the transcription process. The introduction of the SingCrawl data pipeline for large-scale data collection is also a noteworthy contribution, enhancing the model's training data quality and quantity.
The experiments demonstrate VocalParse's state-of-the-art performance across multiple datasets, showcasing its effectiveness in both Automatic Melody Transcription (AMT) and Automatic Lyric Transcription (ALT). The results are robust, with clear metrics provided for evaluation, including Mean Absolute Error (MAE) for melody and Word Error Rate (WER) for lyrics. The ablation studies effectively highlight the importance of the CoT prompting and the SingCrawl pipeline, providing insights into the model's performance drivers.
The paper provides sufficient implementation details, including training configurations and data processing steps. The availability of source code and checkpoints on GitHub enhances reproducibility, although the lack of a demo or interactive component may limit accessibility for some researchers.
The paper acknowledges limitations such as the assumption of a single global tempo for songs, which may not capture variations in performance. Additionally, the model's performance is constrained by the quality of the teacher pipeline used for data annotation. The focus on Mandarin data may also limit generalizability to other languages without further adaptation.
VocalParse has the potential to significantly impact the field of music information retrieval (MIR) and singing voice synthesis (SVS) by providing a scalable solution for automatic singing voice transcription. This could lead to advancements in music generation, annotation, and analysis, facilitating broader applications in music technology and AI-driven creative processes. The main contribution of this paper is the development of VocalParse, a unified and scalable singing voice transcription framework that effectively integrates lyrics and melody transcription using advanced prompting strategies and a novel data collection pipeline. This work represents a significant step forward in addressing the challenges of automatic singing voice transcription, with implications for both academic research and practical applications in music technology.
Music popularity prediction has attracted growing research interest, with relevance to artists, platforms, and recommendation systems. However, the explosive rise of AI-generated music platforms has created an entirely new and largely unexplored landscape, where a surge of songs is produced and consumed daily without the traditional markers of artist reputation or label backing. Key, yet unexplored in this pursuit is aesthetic quality. We propose APEX, the first large-scale multi-task learning framework for AI-generated music, trained on over 211k songs (10k hours of audio) from Suno and Udio, that jointly predicts engagement-based popularity signals - streams and likes scores - alongside five perceptual aesthetic quality dimensions from frozen audio embeddings extracted from MERT, a self-supervised music understanding model. Aesthetic quality and popularity capture complementary aspects of music that together prove valuable: in an out-of-distribution evaluation on the Music Arena dataset, comprising pairwise human preference battles across eleven generative music systems unseen during training, including aesthetic features consistently improves preference prediction, demonstrating strong generalisation of the learned representations across generative architectures.
Primary: Singapore University of Technology and Design
All Institutions: Singapore University of Technology and Design
The main contribution of this paper is the introduction of APEX, a multi-task learning framework that predicts both popularity and aesthetic quality in AI-generated music, demonstrating the complementary relationship between these dimensions and providing a foundation for future research in music recommendation systems. The comprehensive analysis of the technical contributions, methodology, and significance to the field highlights the innovative approach taken and its potential impact on the landscape of music technology.
The proposed APEX framework is a significant advancement in the field of music popularity prediction, particularly for AI-generated music. It employs a multi-task learning approach that integrates aesthetic quality dimensions with engagement-based popularity signals, which is a novel combination in this domain. The use of MERT embeddings for audio representation is well-justified, and the systematic exploration of loss strategies and task configurations demonstrates a rigorous approach to model design. The methodology is comprehensive, addressing both the technical aspects of model training and the theoretical underpinnings of the relationship between aesthetic quality and popularity.
The experiments are robust, utilizing a large-scale dataset of over 211k songs and including a thorough ablation study across 24 experimental conditions. The evaluation on the Music Arena dataset with unseen generative music systems adds significant value, demonstrating the model's generalization capabilities. The results indicate that aesthetic features enhance preference prediction, which is a critical insight for future research in music recommendation systems.
The paper provides detailed implementation details, including dataset construction, embedding extraction, training procedures, and model architectures. The open-source release of the APEX model and its code on GitHub further supports reproducibility, allowing other researchers to validate and build upon this work.
One limitation is the potential bias introduced by the dataset, as it primarily focuses on AI-generated music from specific platforms (Udio and Suno), which may not generalize to all music genres or styles. Additionally, the performance gap on vocal tracks suggests that further refinement is needed for models that incorporate vocal elements, which could be explored in future work.
The implications of this research are substantial, as it addresses the growing domain of AI-generated music and its integration into popular music consumption. By providing a framework that predicts popularity based on intrinsic audio features and aesthetic quality, it opens avenues for improved music recommendation systems and enhances the understanding of how listeners perceive AI-generated music. This work could influence artists, music platforms, and researchers alike, fostering a deeper appreciation for the aesthetic dimensions of music. The main contribution of this paper is the introduction of APEX, a multi-task learning framework that predicts both popularity and aesthetic quality in AI-generated music, demonstrating the complementary relationship between these dimensions and providing a foundation for future research in music recommendation systems. The comprehensive analysis of the technical contributions, methodology, and significance to the field highlights the innovative approach taken and its potential impact on the landscape of music technology.
Training data for bioacoustics is scattered across taxa, regions, and institutions. Centralizing it all is often infeasible. We show that independently fine-tuned BEATs encoders can be composed into a unified 661-species classifier via task vector arithmetic without sharing data. We find that bioacoustic task vectors are near-orthogonal (cosine 0.01-0.09). Their separation aligns closely with spectral distribution distance, a gradient consistent with the acoustic niche hypothesis. This geometry makes simple averaging optimal while sign-conflict methods reduce accuracy by one to six percentage points. Composition also creates an asymmetric gap: species-rich groups lose accuracy relative to joint training while underrepresented taxa gain, a redistribution useful for equitable biodiversity monitoring. We verify linear mode connectivity across all taxonomic pairs, demonstrate zero-shot transfer to new regions, and identify domain negation as a boundary condition where composition fails. These results enable a collaborative paradigm for bioacoustics where institutions share only task vectors to assemble multi-taxa classifiers, preserving data privacy.
Primary: Institute of Science Tokyo
All Institutions: Institute of Science Tokyo, RIKEN BDR
This paper presents a significant advancement in the field of bioacoustics by introducing a novel method for composing multi-taxa classifiers using task vector arithmetic, enabling collaborative model building while preserving data privacy. The combination of ecological principles with machine learning techniques offers a fresh perspective on model merging, with the potential to impact biodiversity monitoring and conservation efforts.
The methodology presented in this paper is innovative, leveraging task vector arithmetic to combine independently fine-tuned models into a unified classifier without sharing data. The authors provide a clear and systematic approach to the problem, including the definition of task vectors and the exploration of their geometric properties in weight space. The use of ecological principles, specifically the acoustic niche hypothesis, to predict the geometry of task vectors is a novel angle that adds depth to the analysis. The methodology is well-structured, with detailed descriptions of the merging strategies and the assumptions made during the process.
The experimental evaluation is robust, with multiple experiments validating the proposed approach. The authors demonstrate linear mode connectivity, analyze task vector geometry, and assess the impact of merging on species classification accuracy. The results are comprehensive, showing both the advantages and limitations of the proposed method. The use of various datasets and the exploration of zero-shot transfer capabilities further strengthen the findings. However, the paper could benefit from more extensive comparisons with existing methods to highlight the advantages of their approach.
The paper provides a thorough account of the experimental setup, including hyperparameters and training protocols, which aids reproducibility. The authors mention using SHA-256 hashes for configuration validation, ensuring that the experiments can be replicated. However, the lack of a detailed description of the datasets used and their preprocessing steps may pose challenges for complete reproducibility.
One limitation of the study is the potential for overfitting to the specific datasets used, which may not generalize to other bioacoustic contexts. Additionally, the assumption of disjoint species sets may not hold in all real-world scenarios, potentially affecting the performance of the proposed method. The authors also acknowledge that domain negation fails, indicating that there are boundaries to the applicability of their approach.
The proposed method has significant implications for biodiversity monitoring and conservation efforts, allowing institutions to collaborate without compromising data privacy. This collaborative paradigm could enhance the development of multi-taxa classifiers, making it easier to monitor diverse ecosystems and respond to conservation needs. The findings could also inspire further research into task vector arithmetic in other domains of machine learning, potentially leading to more efficient and privacy-preserving model training techniques. This paper presents a significant advancement in the field of bioacoustics by introducing a novel method for composing multi-taxa classifiers using task vector arithmetic, enabling collaborative model building while preserving data privacy. The combination of ecological principles with machine learning techniques offers a fresh perspective on model merging, with the potential to impact biodiversity monitoring and conservation efforts.
Music-inspired Automatic Stage Lighting Control (ASLC) has gained increasing attention in recent years due to the substantial time and financial costs associated with hiring and training professional lighting engineers. However, existing methods suffer from several notable limitations: the low interpretability of rule-based approaches, the restriction to single-primary-light control in music-to-color-space methods, and the limited transferability of music-to-controlling-parameter frameworks. To address these gaps, we propose SeqLight, a hierarchical deep learning framework that maps music to multi-light Hue-Saturation-Value (HSV) space. Our approach first customizes SkipBART, an end-to-end single primary light generation model, to predict the full light color distribution for each frame, followed by hybrid Imitation Learning (IL) techniques to derive an effective decomposition strategy that distributes the global color distribution among individual lights. Notably, the light decomposition module can be trained under varying venue-specific lighting configurations using only mixed light data and no professional demonstrations, thereby flexibly adapting across diverse venues. In this stage, we formulate the light decomposition task as a Goal-Conditioned Markov Decision Process (GCMDP), construct an expert demonstration set inspired by Hindsight Experience Replay (HER), and introduce a three-phase IL training pipeline, achieving strong generalization capability. To validate our IL solution for the proposed GCMDP, we conduct a series of quantitative analysis and human study. The code and trained models are provided at https://github.com/RS2002/SeqLight .
Primary: The Hong Kong University of Science and Technology
All Institutions: The Hong Kong Polytechnic University, The University of Hong Kong, City University of Hong Kong
The main contribution of this paper is the introduction of SeqLight, a novel hierarchical framework for automatic stage lighting control that effectively combines music analysis with advanced machine learning techniques to enhance the quality and adaptability of lighting in live performances. This work represents a significant advancement in the intersection of machine learning and performing arts, addressing practical challenges while providing a robust methodological framework.
The paper introduces SeqLight, a hierarchical deep learning framework that innovatively combines a modified Skip-BART model with imitation learning techniques to address the challenges of automatic stage lighting control. The methodology is well-structured, separating the problem into two stages: predicting light distributions and decomposing these into individual light controls via a Goal-Conditioned Markov Decision Process (GCMDP). The use of hybrid imitation learning, particularly the incorporation of Hindsight Experience Replay (HER) and Adversarial Inverse Reinforcement Learning (AIRL), is a notable strength, as it allows for effective learning without the need for extensive expert demonstrations. The approach is comprehensive, addressing both the technical challenges of multi-light control and the practical limitations of data collection in diverse venues.
The experiments are robust, utilizing both quantitative metrics (L1 distance, Wasserstein distance, etc.) and qualitative assessments through human evaluations. The results demonstrate that SeqLight outperforms competitive baselines, including Skip-BART and rule-based methods, in various music styles. The inclusion of a human study adds significant value, providing insights into user preferences and the system's generalization capabilities across different music genres. However, the paper could benefit from clearer presentation of experimental setups and results, particularly in terms of dataset descriptions and evaluation metrics.
The authors provide a GitHub repository with code and trained models, which is a positive aspect for reproducibility. However, the paper lacks detailed descriptions of the datasets used and the specific configurations for training, which could hinder full reproducibility. More explicit guidelines on running the experiments and the environment setup would enhance this aspect.
The paper acknowledges the limitations of its approach, particularly regarding the reliance on mixed light data without professional demonstrations. While the method shows promise in diverse venues, the generalization to real-world scenarios may still face challenges due to the variability in lighting setups and music styles. Additionally, the performance in out-of-domain settings, while improved, still shows some degradation compared to in-domain results.
The proposed method has significant implications for the fields of live music performance and event production, potentially reducing the need for professional lighting engineers and making stage lighting more accessible to amateurs. The ability to adapt to various venues and music styles opens up new avenues for creative expression in live performances. Furthermore, the integration of machine learning in artistic domains like lighting design could inspire further research and applications in related fields. The main contribution of this paper is the introduction of SeqLight, a novel hierarchical framework for automatic stage lighting control that effectively combines music analysis with advanced machine learning techniques to enhance the quality and adaptability of lighting in live performances. This work represents a significant advancement in the intersection of machine learning and performing arts, addressing practical challenges while providing a robust methodological framework.
Sound Event Detection (SED) plays a vital role in audio understanding, with applications in surveillance, smart cities, healthcare, and multimedia indexing. However, conventional SED systems operate under a closed-world assumption, limiting their effectiveness in real-world environments where novel acoustic events frequently emerge. Inspired by the success of open-world learning in computer vision, we introduce the Open-World Sound Event Detection (OW-SED) paradigm, where models must detect known events, identify unseen ones, and incrementally learn from them. To tackle the unique challenges of OW-SED, such as overlapping and ambiguous events, we propose a 1D Deformable architecture that leverages deformable attention to adaptively focus on salient temporal regions. Furthermore, we design a novel Open-World Deformable Sound Event Detection Transformer (WOOT) framework incorporating feature disentanglement to separate class-specific and class-agnostic representations, together with a one-to-many matching strategy and a diversity loss to enhance representation diversity. Experimental results demonstrate that our method achieves marginally superior performance compared to existing leading techniques in closed-world settings and significantly improves over existing baselines in open-world scenarios.
Primary: VNU University of Engineering and Technology
All Institutions: VNU University of Engineering and Technology, Artificial Intelligence Research Center, VNU Information Technology Institute
The main contribution of this paper is the introduction of the OW-SED paradigm and the WOOT framework, which significantly advances the field of sound event detection by enabling models to detect known events, identify unseen ones, and incrementally learn from them. The methodology and experimental results demonstrate a strong technical contribution that addresses real-world challenges in audio understanding.
The paper introduces a novel Open-World Sound Event Detection (OW-SED) paradigm, which is a significant shift from traditional closed-world approaches. The proposed 1D Deformable architecture leverages deformable attention mechanisms to focus on salient temporal regions, addressing the unique challenges posed by overlapping and ambiguous sound events. The introduction of the WOOT framework, which incorporates feature disentanglement and a two-stage training strategy, is innovative and effectively enhances the model's ability to generalize to unseen classes while mitigating catastrophic forgetting. The methodology is well-structured, with clear explanations of the architecture and training processes.
The experimental results demonstrate that the proposed methods outperform existing state-of-the-art techniques in both closed-world and open-world settings. The authors provide comprehensive evaluations using two datasets (URBAN-SED and DESED), showcasing significant improvements in performance metrics such as U-Recall and F1 scores. The experiments are robust, with multiple random seeds used to ensure reliability, and the results are well-presented in comparative tables.
The paper includes detailed implementation details, including the architecture, training protocols, and hyperparameters. However, the absence of a public code repository or demo URL limits the reproducibility of the results. The authors should consider making their code available to facilitate further research and validation of their findings.
One limitation is the reliance on human annotation for labeling unknown events, which may introduce subjectivity and variability in the training process. Additionally, while the model shows strong performance, the paper does not extensively discuss the computational efficiency or scalability of the proposed framework in real-world applications.
The OW-SED paradigm has significant implications for various applications, including surveillance, smart cities, and healthcare, where the ability to detect and learn from novel sound events in dynamic environments is crucial. This work paves the way for more adaptive audio understanding systems that can continuously evolve with their environments. The main contribution of this paper is the introduction of the OW-SED paradigm and the WOOT framework, which significantly advances the field of sound event detection by enabling models to detect known events, identify unseen ones, and incrementally learn from them. The methodology and experimental results demonstrate a strong technical contribution that addresses real-world challenges in audio understanding.
MiniMind-O is an open 0.1B-scale omni model built on the MiniMind language model. It accepts text, speech, and image inputs, and returns both text and streaming speech. The release includes model code, checkpoints, and the main Parquet training datasets for text-to-audio, image-to-text, and audio-to-audio training, making the complete interaction loop directly inspectable. The model uses a full MiniMind backbone as the Thinker and an independent four-layer Talker made from MiniMind blocks. Frozen SenseVoice-Small and SigLIP2 encoders provide speech and image features, which are mapped by lightweight MLP projectors and injected at modality-placeholder positions. The Talker reads a middle-layer Thinker state together with an autoregressive eight-layer Mimi-code buffer. Speaker control is handled by a dedicated speaker token, right-aligned reference codec prompts, and precomputed CAM++ speaker embeddings, so voice conditioning remains part of the audio-code context rather than a separate TTS module. With a 768-dimensional Talker, the dense and MoE variants reach average CERs of 0.0897 and 0.0900 in Thinker--Talker consistency evaluation, with overall voice-cloning similarities of 0.5995 and 0.5937. Beyond reporting a working system, the paper identifies three scale-critical design choices for small omni models: middle-layer semantic bridging, a released multimodal sequence format, and a parameter-efficient eight-codebook interface.
Primary: Independent Researcher
All Institutions: Independent Researcher
The main contribution of this paper is the introduction of MiniMind-O, a small-scale omni model that effectively integrates text, speech, and image modalities while maintaining a focus on reproducibility and parameter efficiency. This work represents a meaningful step towards creating accessible and controllable multimodal systems, with implications for various applications in machine learning and human-computer interaction.
The methodology employed in MiniMind-O is innovative, leveraging a compact architecture that integrates text, speech, and image modalities within a 0.1B parameter framework. The separation of the Thinker and Talker components allows for a more efficient processing pipeline, while the use of middle-layer semantic bridging and low-rank codebook interfaces enhances parameter efficiency. The decision to release the training datasets and model code further promotes transparency and reproducibility in research.
The experimental evaluation is robust, with clear metrics for assessing consistency between the Thinker and Talker outputs. The use of Character Error Rate (CER) and voice-cloning similarity scores provides a quantitative basis for evaluating performance. However, the evaluation metrics focus primarily on consistency rather than subjective quality measures, which could limit the understanding of the model's performance in real-world applications.
The paper emphasizes reproducibility by providing detailed descriptions of the model architecture, training pipeline, and the datasets used. The release of both the model code and training datasets is a significant step towards enabling other researchers to replicate the results. However, the lack of a specific institution may raise questions about the long-term support and maintenance of the project.
The main limitations include the model's performance in generating natural-sounding speech, especially for longer responses, which may not match the quality of larger models. Additionally, the visual pathway relies on a frozen encoder, which may not capture the full complexity of visual inputs. The narrow evaluation focus on consistency may overlook other important aspects of model performance, such as user experience and adaptability.
The potential applications of MiniMind-O are significant, particularly in areas requiring multimodal interaction, such as virtual assistants, educational tools, and accessibility technologies. By providing an open-source framework, the work encourages further research and development in the field of speech-native omni models, potentially leading to advancements in human-computer interaction. The main contribution of this paper is the introduction of MiniMind-O, a small-scale omni model that effectively integrates text, speech, and image modalities while maintaining a focus on reproducibility and parameter efficiency. This work represents a meaningful step towards creating accessible and controllable multimodal systems, with implications for various applications in machine learning and human-computer interaction.
Self-supervised speech models (S3Ms) achieve strong downstream performance, yet their learned representations remain poorly understood under natural and adversarial perturbations. Prior studies rely on representation similarity or global dimensionality, offering limited visibility into local geometric changes. We ask: how do perturbations deform local geometry, and do these shifts track downstream automatic speech recognition (ASR) degradation? To address this, we present GRIDS, a framework using Local Intrinsic Dimensionality (LID) across layer-wise representations in WavLM and wav2vec 2.0. We find that LID increases for all low signal-to noise ratio (SNR) perturbations and diverges at high SNR: benign noise converges toward the clean profile, while adversarial inputs retain early-layer LID elevation. We show LID elevation co-occurs with increased WER, and that layer-wise LID features enable anomaly detection (AUROC 0.78-1.00), opening the door to transcript-free monitoring in S3Ms.
Primary: University of Melbourne
All Institutions: University of Melbourne, Monash University, Johns Hopkins University
The paper presents a comprehensive and innovative framework for analyzing the geometric properties of learned representations in self-supervised speech models, contributing valuable insights into the robustness of these models under various perturbations. The methodology effectively links local geometric changes to performance degradation, marking a significant advancement in the understanding of S3Ms.
The paper introduces a novel framework, GRIDS, which utilizes Local Intrinsic Dimensionality (LID) to analyze the geometric properties of learned representations in self-supervised speech models (S3Ms) under various perturbations. The methodology is well-structured, employing a layer-wise analysis that captures local changes in representation geometry, which is a significant advancement over traditional global measures. The approach effectively links geometric shifts to downstream performance metrics, specifically word error rate (WER), and provides a robust mechanism for anomaly detection without reliance on ground-truth transcripts. The use of k-nearest neighbors (kNN) for LID estimation is appropriate, although the choice of neighborhood size could introduce variability in results.
The experiments are comprehensive, utilizing a well-defined dataset (LibriSpeech) and a variety of perturbation types (benign and adversarial) under controlled signal-to-noise ratio (SNR) conditions. The results demonstrate a clear correlation between LID changes and ASR degradation, reinforcing the framework's validity. The performance metrics, including AUROC for anomaly detection, indicate strong results, particularly for the WavLM model. However, the paper could benefit from additional comparative analysis against existing methods to further establish the effectiveness of GRIDS.
The paper provides detailed descriptions of the experimental setup, including the generation of perturbations and the evaluation protocols. However, the lack of publicly available code or data limits reproducibility. Providing access to the GRIDS framework and datasets would enhance the ability of other researchers to validate and build upon these findings.
The study is limited to specific self-supervised speech models (WavLM and wav2vec 2.0) and does not explore the implications of the findings on other architectures or tasks. Additionally, the focus on layer-wise analysis may overlook global representation dynamics that could be informative. The anomaly detection performance decreases at higher SNRs, suggesting that the method may not be universally applicable across all conditions.
The findings have significant implications for the robustness and interpretability of self-supervised speech models, particularly in real-world applications where adversarial attacks and noise are prevalent. The ability to monitor representation geometry could lead to improved robustness in automatic speech recognition systems and other related fields, such as speaker verification and emotion recognition. The paper presents a comprehensive and innovative framework for analyzing the geometric properties of learned representations in self-supervised speech models, contributing valuable insights into the robustness of these models under various perturbations. The methodology effectively links local geometric changes to performance degradation, marking a significant advancement in the understanding of S3Ms.
Vocal hyperfunction (VH) is a prevalent voice disorder whose ambulatory detection remains challenging despite extensive daily voice data. Prior approaches capture week-long neck-surface accelerometer recordings but collapse them into fixed-length subject-level feature vectors, discarding within-day temporal dynamics encoding nuanced voicing feature interactions. We introduce a novel hybrid architecture combining gradient-boosted trees on day-level distributional features with a CNN-based multiple instance learning (MIL) framework that preserves and learns from from temporal dynamics throughout each day. On the held-out test set, our model exceeds the challenge baselines (AUC: 0.82 PVH, 0.77 NPVH), achieving AUCs of 0.879 for PVH (Rank 5) and 0.848 for NPVH (Rank 3), while also providing insights into clinically relevant information about both pathologies.
Primary: Harvard University
All Institutions: Harvard University, Eaton Peabody Laboratories, Massachusetts Eye and Ear Infirmary
The paper presents a novel hybrid architecture for detecting vocal hyperfunction using attention-based multiple instance learning, significantly advancing the state of the art in ambulatory voice monitoring. The methodology effectively addresses the limitations of previous approaches by preserving temporal dynamics, leading to improved diagnostic accuracy and insights into voice disorders.
The paper introduces a hybrid architecture that effectively combines gradient-boosted trees with a CNN-based multiple instance learning (MIL) framework, which is innovative in the context of ecological momentary assessment for voice disorders. The dual-representation framework allows the model to leverage both global distributional features and local temporal dynamics, addressing the shortcomings of previous methods that relied on fixed-length feature vectors. The attention mechanism in the CNN-MIL architecture is particularly noteworthy, as it enables the model to learn which time segments are most discriminative for vocal hyperfunction detection.
The experiments are robust, utilizing the NeckVibe Challenge dataset, which is the largest publicly available dataset for this task. The reported AUC scores demonstrate significant improvements over existing baselines, particularly for the more challenging non-structural vocal hyperfunction (NPVH) classification. The systematic ablation studies provide strong evidence for the contributions of each model component, confirming the effectiveness of the proposed ensemble strategy.
The paper provides detailed implementation information, including hyperparameters and software versions used, which enhances reproducibility. However, the absence of a publicly accessible code repository limits the ability for other researchers to replicate the results fully.
The study acknowledges that the CNN-MIL framework processes each day independently, which may overlook week-level trends in vocal behavior. Additionally, while the attention mechanism offers some interpretability, the complexity of the model may hinder full understanding of the feature interactions. Future work could explore hierarchical models and causal inference approaches.
The findings have significant implications for the field of voice disorder diagnostics, particularly in enhancing ambulatory monitoring techniques. By capturing temporal dynamics, the proposed approach could lead to more accurate and timely interventions for individuals suffering from vocal hyperfunction, potentially improving patient outcomes. The paper presents a novel hybrid architecture for detecting vocal hyperfunction using attention-based multiple instance learning, significantly advancing the state of the art in ambulatory voice monitoring. The methodology effectively addresses the limitations of previous approaches by preserving temporal dynamics, leading to improved diagnostic accuracy and insights into voice disorders.
Recent advances in emotional voice conversion (EVC) have enabled the generation of expressive synthetic speech, raising new concerns in audio deepfake detection. Existing approaches treat speech as a homogeneous signal and largely overlook its internal phonetic structure, limiting their interpretability in emotionally conditioned settings. In this work, we propose a phoneme-level framework to analyze emotionally manipulated synthetic speech using real and EVC-generated speech under matched emotional conditions with shared transcripts, phoneme-aligned TextGrids, and WavLM-based embeddings. Our results show that phoneme behavior varies across categories, with complex vowels and fricatives exhibiting higher divergence while simpler phonemes remain more stable. Phonemes with larger distributional differences are also found to be more easily detected, consistently across multiple emotions and synthesis systems. These findings demonstrate that phoneme-level analysis is an effective and interpretable approach for detecting emotionally manipulated synthetic speech.
Primary: Wichita State University
All Institutions: Wichita State University, Institut national de la recherche scientifique (INRS--EMT), INRS-UQO Mixed Research Unit on Cybersecurity
This paper presents a novel phoneme-level framework for detecting emotionally manipulated synthetic speech, demonstrating that phoneme behavior varies significantly across categories and can be effectively analyzed for deepfake detection. The methodology is sound, the experimental design is robust, and the results contribute valuable insights to the field of audio deepfake detection.
The proposed phoneme-level framework is innovative, addressing the limitations of existing deepfake detection methods that overlook phonetic structures in emotionally manipulated speech. The use of WavLM embeddings and Kullback-Leibler divergence for phoneme-level analysis is a strong methodological choice, allowing for nuanced insights into how emotional voice conversion affects phoneme characteristics. The controlled dataset and phoneme-aligned TextGrids enhance the robustness of the analysis.
The experiments are well-structured, utilizing a curated dataset that allows for direct comparisons between real and synthetic speech across multiple emotions. The results demonstrate clear patterns in phoneme sensitivity to emotional manipulation, with statistical analyses supporting the findings. The use of SVM for classification and the correlation analysis between KLD and classification accuracy are appropriate and yield meaningful insights.
The paper mentions the release of a curated dataset with aligned transcripts and phoneme-level annotations, which supports reproducibility. However, specific implementation details regarding the SVM training and evaluation metrics could be more explicitly stated to facilitate replication.
One limitation is the focus on only two EVC systems and a limited number of speakers, which may restrict the generalizability of the findings. Additionally, the paper does not explore the potential impact of background noise or other real-world conditions on detection performance.
The findings have significant implications for audio deepfake detection, particularly in high-stakes environments such as journalism and legal proceedings. By providing a phoneme-level analysis, the work contributes to the development of more interpretable and robust detection systems, potentially enhancing trust in audio communications. This paper presents a novel phoneme-level framework for detecting emotionally manipulated synthetic speech, demonstrating that phoneme behavior varies significantly across categories and can be effectively analyzed for deepfake detection. The methodology is sound, the experimental design is robust, and the results contribute valuable insights to the field of audio deepfake detection.
Tibetan text-to-speech (TTS) has long been challenged by scarce speech resources, significant dialectal variation, and the complex mapping between written text and spoken pronunciation. To address these issues, this work presents, to the best of our knowledge, the first large-model-based Tibetan TTS system in the industry, built upon a large speech synthesis model developed by Xingchen AGI Lab. The proposed system integrates data quality enhancement, Tibetan-oriented text representation and tokenizer adaptation, and cross-lingual adaptive training for low-resource Tibetan speech synthesis. Experimental results show that the system can generate stable, natural, and intelligible Tibetan speech under low-resource conditions. In subjective evaluation, the MOS scores of the syllable-level and BPE-based systems reach 4.28 and 4.35, while their pronunciation accuracies reach 97.6% and 96.6%, respectively, outperforming an external commercial Tibetan TTS interface. These results demonstrate that combining a large-model backbone with Tibetan-oriented text representation adaptation and cross-lingual adaptive training enables highly usable low-resource Tibetan speech synthesis, and also provides a technical foundation for future unified multi-dialect Tibetan speech synthesis.
Primary: Xingchen AGI Lab
All Institutions: Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd, Xizang University, Qinghai Normal University, University of Electronic Science and Technology of China
This paper establishes a pioneering framework for low-resource Tibetan speech synthesis, combining innovative methodologies with practical applications. The integration of data quality enhancement, tailored text representation, and cross-lingual training marks a significant advancement in the field of speech synthesis, particularly for minority languages facing resource constraints.
The paper presents a comprehensive approach to Tibetan TTS by integrating a data quality enhancement pipeline, a Tibetan-oriented text representation, and a cross-lingual adaptive training strategy. The methodology is well-structured, addressing the unique challenges of Tibetan speech synthesis, such as dialectal variation and resource scarcity. The use of a large-model backbone and the adaptation of tokenization strategies specifically for Tibetan linguistic characteristics are notable innovations that enhance the model's performance under low-resource conditions.
The experiments are robust, with both subjective (MOS scores) and objective (pronunciation accuracy) evaluations demonstrating the effectiveness of the proposed system. The reported MOS scores of 4.28 and 4.35 for the syllable-level and BPE-based systems, respectively, indicate a high level of naturalness and intelligibility. The comparison with an external commercial TTS interface further validates the system's performance, showcasing its potential for practical applications.
The paper lacks detailed implementation specifics, such as code availability or dataset access, which could hinder reproducibility. While the methodology is clearly described, the absence of a public repository or demo limits the ability for other researchers to replicate the results.
The study primarily focuses on the U-Tsang dialect, which may limit the generalizability of the findings to other Tibetan dialects. Additionally, while the proposed methods show promise, the reliance on a large pretrained model may not be feasible in all low-resource scenarios, particularly where such models are not available.
The development of a Tibetan TTS system has significant implications for cultural preservation, education, and accessibility in Tibetan-speaking regions. By providing a framework for low-resource language synthesis, this work could serve as a model for similar efforts in other underrepresented languages, promoting linguistic diversity in technology. This paper establishes a pioneering framework for low-resource Tibetan speech synthesis, combining innovative methodologies with practical applications. The integration of data quality enhancement, tailored text representation, and cross-lingual training marks a significant advancement in the field of speech synthesis, particularly for minority languages facing resource constraints.
Periodic patterns are fundamental cues in multimedia signals and systems, including repetitive motion in video (e.g., gait cycles), rhythmic and pitch-related structure in audio, and recurring textures in image sequences. When such user-generated streams are collected from edge devices, local differential privacy (LDP) is appealing because it perturbs data before upload; however, the injected noise can corrupt spectral peaks and induce phase drift, making period estimation unreliable and degrading reconstruction quality. We propose \textbf{CPR} (\textit{Cycle and Phase Recovery}), a period-aware reconstruction framework for periodic time series under LDP. CPR performs multi-scale period probing and multi-consensus selection to suppress noise-induced spectral interference, then aggregates perturbed samples at matched within-cycle phase positions to stabilize phase alignment across cycles. To recover the underlying per-phase values, CPR combines EM-based denoising with kernel density estimation, improving robustness under tight privacy budgets. Experiments on two real-world periodic datasets demonstrate that CPR better preserves periodic structure and consistently achieves lower reconstruction error than representative LDP baselines, especially in the low-$ε$ regime.
Primary: Taiyuan University of Technology
All Institutions: Taiyuan University of Technology, University of Michigan, Ann Arbor
The main contribution of this paper is the development of the CPR framework, which effectively addresses the challenges of reconstructing periodic time series under local differential privacy. This work significantly advances the field by providing a robust solution that preserves critical periodic structures while ensuring privacy, thus opening new avenues for research and application in multimedia signal processing.
The proposed CPR framework introduces a novel approach to reconstruct periodic time series under local differential privacy (LDP) by addressing the specific challenges posed by noise-induced spectral interference and phase drift. The methodology is robust, employing multi-scale period probing and phase-aware aggregation, which are well-justified in the context of multimedia signals. The integration of EM-based denoising with kernel density estimation is a significant technical contribution, enhancing the reconstruction quality while maintaining privacy. The paper effectively articulates the rationale behind each methodological choice and demonstrates a clear understanding of the underlying challenges.
The experiments conducted on real-world datasets are comprehensive and well-structured, showcasing the effectiveness of CPR compared to several baseline methods. The evaluation metrics, particularly the cosine distance for reconstruction accuracy, are appropriate for the task. The results convincingly demonstrate that CPR outperforms existing methods, especially under tight privacy constraints, which is crucial for practical applications. However, the paper could benefit from a more detailed discussion of the experimental setup, including hyperparameter choices and potential variations in results across different datasets.
The paper provides a clear description of the experimental setup and methodologies used, but it lacks specific implementation details that would facilitate reproducibility. Key parameters and configurations for the experiments are mentioned, yet the absence of a publicly available code repository limits the ability for others to replicate the results. Including a link to a GitHub repository or similar would greatly enhance reproducibility.
One limitation of the study is the reliance on specific datasets that may not fully represent the diversity of periodic signals encountered in real-world applications. Additionally, while the proposed method shows significant improvements, the paper does not address the computational complexity of the CPR framework, which may affect its applicability in resource-constrained environments. The potential trade-offs between privacy and reconstruction accuracy under varying conditions could also be explored further.
The implications of this research are significant, particularly in fields where privacy-preserving data collection is paramount, such as healthcare and personal monitoring systems. By enabling accurate reconstruction of periodic signals while adhering to strict privacy constraints, this work has the potential to enhance the usability of sensitive data in various applications, including motion analysis and behavioral monitoring. The findings may also inspire further research into privacy-aware signal processing techniques. The main contribution of this paper is the development of the CPR framework, which effectively addresses the challenges of reconstructing periodic time series under local differential privacy. This work significantly advances the field by providing a robust solution that preserves critical periodic structures while ensuring privacy, thus opening new avenues for research and application in multimedia signal processing.
We study example-level private supervised speech classification under a practical release constraint: training may access privileged side information, but the released model must be audio-only. This setting is important because speech systems can often exploit richer side information during development, whereas deployment and release require a lightweight unimodal model with auditable privacy guarantees. Using DP-SGD on the private dataset $D_{\text{priv}}$, we identify a strong-privacy failure mode ($ε\le 1$) on imbalanced tasks, where training may collapse to a near single-class predictor, a phenomenon that overall accuracy can obscure. We therefore emphasize Macro-F1, balanced accuracy, and a simple collapse diagnostic. This failure is especially problematic in our release setting because a collapsed private teacher cannot provide useful supervision for the downstream audio-only student. To address this setting under strong privacy, we propose a two-stage protocol: (i) train a (possibly multimodal) DP teacher on $D_{\text{priv}}$, and (ii) distill an audio-only student on a fixed, recording-disjoint auxiliary dataset $D_{\text{aux}}$ using one-shot offline teacher probability outputs, releasing only the student. The DP guarantee applies only to $D_{\text{priv}}$; we make no DP claim for $D_{\text{aux}}$, and privacy of the released student with respect to $D_{\text{priv}}$ follows by post-processing. We frame this setting as involving four coupled bottlenecks: speech-induced optimization instability under DP-SGD, minority-class erosion under clipping and noise, teacher over-reliance on privileged modalities unavailable at deployment, and train--deploy modality mismatch. We address them with a DP-stabilizing acoustic front-end (DSAF), minibatch-adaptive bounded loss reweighting (AW-DP), privileged-modality dropout, and offline teacher-to-student distillation.
Primary: Taiyuan University of Technology
All Institutions: Shanxi Key Laboratory of Industrial Internet Security, University of Michigan, Taiyuan University of Technology
The main contribution of this paper is a novel two-stage protocol for private speech classification that mitigates prediction collapse and class imbalance while ensuring differential privacy. This work significantly advances the field of privacy-preserving machine learning by providing effective solutions to critical challenges in deploying speech classification systems.
The paper presents a novel two-stage protocol for private speech classification that effectively addresses the challenges of differential privacy (DP) in imbalanced datasets. The methodology is well-structured, incorporating a DP teacher trained on private data followed by offline distillation to create an audio-only student model. The authors introduce several innovative techniques such as the DP-Stabilizing Acoustic Front-End (DSAF) and Imbalance-aware Weighted DP-SGD (AW-DP) to mitigate issues related to prediction collapse and class imbalance. The use of privileged-modality dropout further enhances the robustness of the model by discouraging reliance on privileged information during deployment. Overall, the methodology is comprehensive and addresses critical bottlenecks in private speech classification.
The experiments are thorough, utilizing the Mozilla Common Voice dataset to evaluate the proposed methods. The authors provide a clear comparison of their two-stage distillation approach against a single-stage DP audio-only model. The results demonstrate significant improvements in Macro-F1 and balanced accuracy metrics, particularly under strong privacy constraints, highlighting the effectiveness of their approach. The inclusion of various metrics to diagnose collapse, such as Maj-Pred, adds depth to the evaluation. However, further exploration of the impact of different auxiliary dataset sizes could enhance the robustness of the findings.
The paper provides sufficient details regarding the experimental setup, including the use of Python and PyTorch for implementation. However, the absence of a public code repository or demo URL limits reproducibility. Future work should consider making the code available to facilitate validation of results and encourage further research in this area.
One limitation of the study is the lack of a comprehensive evaluation of the model's performance across diverse datasets beyond the Mozilla Common Voice dataset. Additionally, while the paper addresses the issue of prediction collapse, it does not fully explore the implications of using auxiliary datasets that may not be representative of the deployment environment. The privacy guarantee for the auxiliary dataset is also not claimed, which could raise concerns about potential data leakage.
The proposed methods have significant implications for the deployment of speech classification systems in privacy-sensitive applications, such as voice assistants and transcription services. By ensuring that models can be trained with rich side information while maintaining privacy in the released artifacts, this work contributes to the development of more secure and trustworthy AI systems. The approach could be extended to other modalities and applications, enhancing the overall impact on the field of machine learning and privacy-preserving technologies. The main contribution of this paper is a novel two-stage protocol for private speech classification that mitigates prediction collapse and class imbalance while ensuring differential privacy. This work significantly advances the field of privacy-preserving machine learning by providing effective solutions to critical challenges in deploying speech classification systems.
We present the Streaming Reservoir Convergence Theorem (SRCT), a novel mathematical framework for multi-provider adaptive bitrate streaming that addresses three fundamental structural weaknesses in current systems: linear provider probing, reactive failover, and cold standby transitions. SRCT models stream acquisition as a concurrent reservoir filling problem$-$probing all $N$ providers simultaneously rather than in batches$-$and maintains $k$ pre-verified, pre-fetched standby streams alongside the active stream to enable sub-second failover with zero user-visible disruption. We prove four principal results: (1) a harmonic lower bound on reservoir safety showing that $k$ independent streams provide $H_k / \barλ$ expected uptime where $H_k$ is the $k$-th harmonic number; (2) a concurrent acquisition speedup $S(N,b) = (N/b) \cdot (1-F^b)/(1-F^N)$ over batched probing, yielding $3$-$5\times$ practical improvement; (3) monotonic non-decreasing quality under lazy-refill with convergence to the Pareto-optimal frontier; and (4) a prospect-weighted switching rule$-$using Kahneman-Tversky value functions with $α=β=0.88$, $λ=2.25$ $-$ that provably eliminates thrashing between similar-quality streams via a no-thrash bound on the expected switch count. We implement SRCT across two production streaming pipelines: a primary movie/TV system serving 12+ HLS providers with $k=3$ reservoir slots, and a live sports system with multi-format DASH/HLS failover. Empirical verification via Monte Carlo simulation (5000 trials) confirms all four theorems across 22 independent checks. The reservoir of $k=3$ streams achieves $9.15\times$ mean time to depletion versus a single stream, and concurrent probing of 12 providers at 40% failure rate yields a $4.27\times$ speedup over the current batched-by-3 default.
Primary: Sperix Labs
All Institutions: Sperix Labs, KNUST
The main contribution of this paper is the introduction of the Streaming Reservoir Convergence Theorem, which provides a comprehensive framework for improving multi-provider adaptive streaming. The technical contributions, including theoretical proofs and empirical validation, position this work as a notable advancement in the field of audio streaming and adaptive bitrate technologies.
The paper introduces the Streaming Reservoir Convergence Theorem (SRCT), which presents a novel mathematical framework for adaptive bitrate streaming across multiple providers. The methodology is robust, employing a combination of theoretical proofs and practical implementations. The authors effectively unify several aspects of streaming (provider probing, failover, and quality selection) into a single reservoir model, which is a significant advancement over traditional methods. The use of prospect theory to inform the switching rules adds a unique psychological dimension to the algorithm, enhancing its applicability in real-world scenarios.
The experimental evaluation is thorough, with empirical verification conducted through Monte Carlo simulations and deterministic checks. The results demonstrate significant improvements in mean time to depletion and speedup in stream acquisition, validating the theoretical claims made in the paper. The experiments are well-designed, with a clear connection between the theoretical framework and practical outcomes.
While the paper provides a detailed description of the algorithm and its implementation, it lacks a publicly accessible code repository or demo URL, which would enhance reproducibility. The absence of shared code limits the ability of other researchers to validate the findings independently.
The paper acknowledges the limitations of the conditional independence assumption, which may not hold during large-scale outages. Additionally, the reliance on a Markov model for stream viability may not capture more complex, time-varying availability patterns. The parameters derived from prospect theory are also noted as potentially needing further calibration for specific streaming contexts.
The proposed framework has significant implications for the future of adaptive streaming technologies, particularly in enhancing user experience through reduced buffering and improved quality. The integration of psychological principles into automated decision-making processes may influence other domains beyond streaming, such as network management and real-time systems. The main contribution of this paper is the introduction of the Streaming Reservoir Convergence Theorem, which provides a comprehensive framework for improving multi-provider adaptive streaming. The technical contributions, including theoretical proofs and empirical validation, position this work as a notable advancement in the field of audio streaming and adaptive bitrate technologies.
Recent advances in voice cloning and text-to-speech synthesis have made partial speech manipulation - where an adversary replaces a few words within an utterance to alter its meaning while preserving the speaker's identity - an increasingly realistic threat. Existing audio deepfake detection benchmarks focus on utterance-level binary classification or single-region tampering, leaving a critical gap in detecting and localizing multiple inpainted segments whose count is unknown a priori. We address this gap with three contributions. First, we introduce MIST (Multiregion Inpainting Speech Tampering), a large-scale multilingual dataset spanning 6 languages with 1-3 independently inpainted word-level segments per utterance, generated via LLM-guided semantic replacement and neural voice cloning, with fake content constituting only 2-7% of each utterance. Second, we propose ISA (Iterative Segment Analysis), a backbone-agnostic framework that performs coarse-to-fine sliding-window classification with gap-tolerant region proposal and boundary refinement to recover all tampered regions without prior knowledge of their count. Third, we define SF1@tau, a segment-level F1 metric based on temporal IoU matching that jointly evaluates region count accuracy and localization precision. Zero-shot evaluation reveals that partial inpainting at word granularity remains unsolved by existing deepfake detectors: utterance-level classifiers trained on fully synthesized speech assign near zero fake probability to MIST utterances where only 2-7% of content is manipulated. ISA consistently outperforms non-iterative baselines in this challenging setting, and the dataset, code, and evaluation toolkit are publicly released.
Primary: Posts and Telecommunications Institute of Technology
All Institutions: Posts and Telecommunications Institute of Technology
The paper presents a comprehensive approach to addressing the challenges of multi-region speech inpainting detection, contributing valuable resources and methodologies to the field of audio forensics. The introduction of the MIST dataset and the ISA framework represents a meaningful step forward in the ongoing battle against audio deepfakes and misinformation.
The paper introduces a novel dataset (MIST) designed specifically for multi-region speech inpainting detection, addressing significant gaps in existing benchmarks that focus primarily on single-region tampering. The methodology includes a comprehensive generation pipeline for creating the dataset, which utilizes LLM-guided semantic replacements and neural voice cloning to produce high-quality tampered audio. The proposed Iterative Segment Analysis (ISA) framework is robust and backbone-agnostic, allowing for effective localization of tampered segments without prior knowledge of their count. The introduction of the SF1@tau metric is a significant advancement, providing a more nuanced evaluation of detection performance by accounting for both segment count and localization precision.
The experiments conducted are thorough, employing a zero-shot evaluation strategy that highlights the challenges of detecting partial inpainting. The results demonstrate that ISA outperforms baseline methods in terms of localization accuracy, even when using a classifier not specifically trained on the dataset. The paper includes a detailed breakdown of results by language and variant, providing insights into the performance across different linguistic contexts and manipulation types. However, the overall SF1 scores remain low, indicating that the task is inherently challenging and that further work is needed to improve detection capabilities.
The authors provide sufficient details regarding the dataset generation process, the ISA framework, and the evaluation metrics, which supports reproducibility. The dataset and code are publicly available, which is a positive aspect for the research community. However, the implementation details of the ISA framework could benefit from additional clarity regarding hyperparameter tuning and specific configurations used in experiments.
The paper acknowledges the limitations of existing audio deepfake detection systems, particularly their inability to handle partial inpainting effectively. The ISA method, while innovative, still relies on a backbone classifier that was not trained on the specific task of multi-region tampering, which may limit its performance. Additionally, the low absolute scores in the experiments suggest that the problem remains challenging, and the proposed methods may require further refinement and optimization.
The implications of this research are significant, particularly in the context of audio forensics and misinformation detection. As voice cloning technology advances, the ability to detect and localize tampered speech becomes increasingly critical for maintaining trust in audio communications. The MIST dataset and ISA framework can serve as foundational tools for future research in this area, potentially leading to improved detection methods and better understanding of audio manipulation threats. The paper presents a comprehensive approach to addressing the challenges of multi-region speech inpainting detection, contributing valuable resources and methodologies to the field of audio forensics. The introduction of the MIST dataset and the ISA framework represents a meaningful step forward in the ongoing battle against audio deepfakes and misinformation.
Stage-wise audio-visual encoders propagate fused intermediate states across layers, making the formation of later representations depend on the readiness of earlier fusion states. Strong local audio-visual agreement provides useful correspondence evidence, yet a fused state also needs sufficient cross-layer and cross-modal support before it can reliably guide later fusion. This paper studies this issue through propagation-aware representation readiness and formulates premature perceptual commitment as a readiness-deficiency problem, where local plausibility, propagation influence, and support insufficiency jointly appear at an intermediate stage. We propose the Delayed Perceptual Commitment Network (DPC-Net), an encoder-level framework that estimates an observable readiness-deficiency surrogate, localizes the intervention-sensitive bottleneck, and applies support-aware correction with cross-layer and cross-modal evidence. DPC-Net preserves task-specific heads, losses, decoding modules, and evaluation protocols, making it applicable to different audio-visual tasks through encoder-side intervention. Experiments on audio-visual speech separation, audio-visual event localization, and audio-visual speech recognition show consistent improvements across reconstruction, localization, and recognition regimes. Further analyses on component contribution, selection criteria, counterfactual intervention, and readiness trajectories support the effectiveness of readiness-guided bottleneck correction.
Primary: Lingnan University
All Institutions: Lingnan University, University of Southern Queensland, Wuhan University of Technology, Hong Kong Metropolitan University
The main contribution of this work is the introduction of DPC-Net, a novel framework that enhances representation readiness in audio-visual learning by addressing premature perceptual commitment through a readiness-deficiency approach. This research significantly advances the understanding of audio-visual representation learning, providing a robust mechanism to improve performance across various tasks while preserving the integrity of task-specific architectures.
The paper introduces the Delayed Perceptual Commitment Network (DPC-Net), which innovatively addresses the issue of representation readiness in stage-wise audio-visual learning by formulating premature perceptual commitment as a readiness-deficiency problem. The methodology is well-structured, utilizing a readiness-deficiency surrogate to localize bottlenecks and applying support-aware corrections, thus enhancing the robustness of audio-visual representations. The approach is grounded in theoretical insights from human perception, which strengthens its conceptual foundation.
The experiments are comprehensive, covering three distinct audio-visual tasks: speech separation, event localization, and speech recognition. The results demonstrate consistent improvements across various metrics, indicating the effectiveness of the proposed method. The use of controlled comparisons with baseline models adds rigor to the evaluation, although the paper could benefit from additional qualitative assessments of the generated outputs.
The paper outlines the implementation details, including the architecture and training protocols, which aids reproducibility. However, the absence of a publicly available code repository limits the ability for other researchers to replicate the findings fully.
One limitation is the reliance on specific datasets for evaluation, which may not generalize across all audio-visual tasks. Additionally, the paper does not address potential computational overhead introduced by the DPC-Net framework, which could impact its practical deployment in real-time systems.
The proposed framework has significant implications for improving audio-visual learning systems, particularly in applications requiring robust performance under adverse conditions, such as speech recognition in noisy environments. The insights gained from this research could lead to advancements in multimodal AI systems, enhancing their ability to process and integrate diverse sensory inputs effectively. The main contribution of this work is the introduction of DPC-Net, a novel framework that enhances representation readiness in audio-visual learning by addressing premature perceptual commitment through a readiness-deficiency approach. This research significantly advances the understanding of audio-visual representation learning, providing a robust mechanism to improve performance across various tasks while preserving the integrity of task-specific architectures.
A common design pattern in high-quality music generation is to handle structure and fidelity in different representation spaces: a generator first models high-level structure, followed by diffusion-based or neural decoding stages that reconstruct fine details. In this work, we explore an alternative view: both may be progressively modeled within a single deep acoustic-token hierarchy. To study this, we build a 64-layer residual vector quantization (RVQ) acoustic representation and propose a two-stage coarse-to-fine generation framework. A backbone model first generates coarse acoustic tokens for the full track, and a super-resolution model then completes finer tokens within the same acoustic token space. The super-resolution stage works at full-track scale and refines tokens layer by layer while running in parallel over time, leading to a fixed 62-step inference process. To jointly improve lyric alignment and fine-detail reconstruction, we further introduce hybrid-attention training: the alignment objective uses causal attention, while layer-wise refinement uses full attention. A key finding is that text--vocal alignment can emerge within pure acoustic-token language modeling, without requiring a separate semantic token stage. Moreover, initializing the super-resolution model from the trained backbone significantly improves convergence and final quality. Taken together, our results suggest that high-quality music generation can be effectively pursued without separating structure and fidelity into heterogeneous representation spaces. Instead, both can be progressively modeled within a unified acoustic-token hierarchy, pointing toward a simpler and more unified path to high-quality music generation.
Primary: Central Conservatory of Music
All Institutions: Central Conservatory of Music, Tsinghua University
The main contribution of this paper is the introduction of Khala, a high-fidelity music generation system that effectively models both musical structure and acoustic fidelity within a unified acoustic-token hierarchy, demonstrating competitive performance against existing systems. This work significantly advances the field of music generation by providing a novel methodology that integrates lyric alignment and acoustic detail refinement in a single framework, showcasing the potential for future developments in this area.
The paper presents a novel approach to music generation by using a unified acoustic-token hierarchy, which contrasts with existing methods that typically separate structure and fidelity into different stages. The introduction of a two-stage coarse-to-fine generation framework and hybrid-attention training for lyric alignment is particularly innovative. The methodology is well-structured, with clear explanations of the architecture and training processes, including the use of a 64-layer residual vector quantization (RVQ) acoustic representation. The authors effectively address the challenges of generating coherent and high-fidelity music while maintaining lyric alignment, which is a significant advancement in the field.
The experiments conducted in the paper include a large-scale human preference evaluation, which is crucial for assessing the quality of generated music. The use of both subjective (mean Overall Score and BT-derived Elo) and objective metrics provides a comprehensive evaluation framework. The results indicate that Khala performs competitively against both commercial and open-source systems, showcasing its effectiveness in real-world applications. The ablation studies further validate the importance of the proposed training strategies, particularly the backbone initialization and hybrid-attention training.
The paper provides detailed implementation information, including model architectures, training strategies, and dataset descriptions. The availability of code and model checkpoints on GitHub enhances reproducibility, allowing other researchers to replicate the experiments and build upon the work. However, the paper could benefit from more explicit details on hyperparameter settings and training configurations to facilitate easier reproduction.
One limitation noted is the reliance on a two-stage model design, which, while practical, may not fully exploit the potential of a unified model that integrates both coarse generation and fine-layer refinement. Additionally, the paper acknowledges that the current tokenizer, while effective, could be improved for even higher fidelity, suggesting that future work could focus on enhancing the acoustic representation further.
The work has significant implications for the field of music generation, particularly in developing systems that can produce high-quality music with coherent structure and fidelity without relying on separate semantic stages. This approach could pave the way for more integrated and efficient music generation systems, potentially impacting various applications in entertainment, education, and creative industries. The findings also suggest a promising direction for future research in audio modeling and machine learning. The main contribution of this paper is the introduction of Khala, a high-fidelity music generation system that effectively models both musical structure and acoustic fidelity within a unified acoustic-token hierarchy, demonstrating competitive performance against existing systems. This work significantly advances the field of music generation by providing a novel methodology that integrates lyric alignment and acoustic detail refinement in a single framework, showcasing the potential for future developments in this area.
In this paper, we propose MelShield, a robust, in-generation, keyed audio watermarking framework that embeds identifiable signals into AI-generated audio for copyright protection and reliable attribution. Specifically, MelShield operates in the Mel-spectrogram domain during the generation process, targeting intermediate acoustic representations in Mel-conditioned pipelines for text-to-speech (TTS) generation. The core idea is to treat the intermediate Mel-spectrogram as the host signal and embed a short binary payload via low-energy, keyed spread-spectrum perturbations distributed across carefully selected time-frequency regions prior to waveform synthesis. By performing watermarking before vocoder inference, MelShield remains plug-and-play for Mel-conditioned TTS architectures and does not require modification or retraining of the underlying TTS generation vocoder, such as DiffWave and HiFi-GAN. Moreover, the multi-user keyed construction enables scalable user-specific attribution, while the keyed verification mechanism limits unauthorized decoding, thereby reducing the risk of large-scale extractor probing and adversarial analysis. Extensive experiments on DiffWave and HiFi-GAN demonstrate that MelShield achieves reliable watermark extraction, approaching 100\% bit accuracy, even under signal distortions, e.g., compression and additive noise, while preserving high perceptual audio quality.
Primary: Queen's University
All Institutions: Queen's University, University of Waterloo
MelShield presents a novel in-generation audio watermarking framework that effectively integrates into TTS systems, enhancing copyright protection and attribution mechanisms. The comprehensive evaluation and innovative methodology position this work as a significant contribution to the field of audio processing and machine learning.
The methodology presented in MelShield is innovative, leveraging a keyed spread-spectrum approach for watermarking directly in the Mel-spectrogram domain of TTS systems. This is a significant advancement over traditional post-hoc watermarking methods, as it integrates watermarking seamlessly into the audio generation pipeline without requiring modifications to existing vocoders. The use of low-energy perturbations and adaptive masking to maintain audio quality while embedding watermarks is particularly noteworthy. The authors provide a clear and systematic approach to embedding and extracting watermarks, which is well-justified and theoretically sound.
The experimental evaluation is comprehensive, utilizing two prominent TTS vocoders (DiffWave and HiFi-GAN) and a robust dataset (LJSpeech 1.1). The results demonstrate high bit accuracy for watermark recovery under various conditions, including common signal distortions. The paper effectively compares MelShield against existing watermarking methods, showcasing its superior performance in terms of robustness and fidelity. The use of multiple evaluation metrics (PESQ, STOI, DNSMOS) adds credibility to the results, although the paper could benefit from more extensive user studies to assess perceptual quality in real-world scenarios.
The paper provides a detailed description of the experimental setup, including the datasets, vocoder configurations, and watermark embedding parameters. However, it lacks a publicly accessible code repository or demo URL, which would enhance reproducibility and allow other researchers to validate the findings. Clearer documentation of the implementation would also aid in replicating the experiments.
One limitation is the reliance on specific vocoders, which may not generalize to all TTS systems. While the authors claim model-agnostic deployment, the performance may vary with different architectures not tested in the study. Additionally, the paper does not address potential vulnerabilities to advanced adversarial attacks that could target the watermarking system. The scalability of the approach in high-demand real-world applications remains to be fully explored.
The implications of this work are significant, particularly in the context of copyright protection and attribution for AI-generated audio. As deepfake technologies become more prevalent, robust watermarking solutions like MelShield can help mitigate risks associated with misinformation and unauthorized content distribution. The framework could be applied across various domains, including media production, digital rights management, and content verification systems. MelShield presents a novel in-generation audio watermarking framework that effectively integrates into TTS systems, enhancing copyright protection and attribution mechanisms. The comprehensive evaluation and innovative methodology position this work as a significant contribution to the field of audio processing and machine learning.
Generating expressive conducting gestures from music is a challenging cross-modal motion synthesis problem: the output must follow long-range musical structure, preserve beat-level synchronization, and remain plausible as a fine-grained 3D human performance. Existing conducting-motion studies are often limited by sparse pose representations, small-scale data, or evaluation protocols that do not directly measure whether music and gesture are mutually aligned. This paper presents TransConductor, a Transformer-based framework for music-driven conducting gesture generation. We introduce ConductorMotion, a SMPL-parameter data construction pipeline that recovers detailed body motion from conducting videos and forms a dataset targeted at professional conducting gestures. Given acoustic descriptors extracted from audio and an initial pose, TransConductor uses a Trans-Temporal Music Encoder and a Trans-Temporal Conducting Gesture Decoder to autoregressively predict SMPL pose parameters. To better assess artistic correspondence, we further build a retrieval-based evaluation model that embeds music and gestures into a shared space and yields FID, modality distance, multi-modality distance, and diversity metrics. Experiments show that TransConductor outperforms dance-generation and conducting-generation baselines, while ablations verify the benefits of the Transformer backbone and the proposed alignment loss.
Primary: Beijing Jiaotong University
All Institutions: Beijing Jiaotong University, Malou Tech Inc, South-Central Minzu University, Fudan University, Renmin University of China
This paper presents a significant advancement in the field of music-driven motion synthesis through the introduction of a Transformer-based framework for generating conducting gestures. The methodology effectively combines detailed pose representation with a novel evaluation approach, setting a new standard for future research in this area.
The proposed methodology introduces a novel Transformer-based framework, TransConductor, which effectively addresses the challenge of generating conducting gestures from music. The use of SMPL parameters for detailed pose representation is a significant advancement over traditional sparse keypoint methods, allowing for a more nuanced and expressive depiction of conducting motions. The dual encoder-decoder architecture, comprising a Trans-Temporal Music Encoder and a Trans-Temporal Conducting Gesture Decoder, is well-conceived, leveraging the strengths of self-attention mechanisms to capture long-range dependencies in both music and gesture. The introduction of a retrieval-based evaluation model further enhances the methodology by providing a more meaningful assessment of the artistic correspondence between music and gestures, which is often overlooked in traditional metrics.
The experimental evaluation is robust, comparing the proposed model against established baselines in dance and conducting generation. The reported metrics (FID, M-Dist, MM-Dist, and diversity) indicate significant improvements in the quality and alignment of generated gestures with the corresponding music. The ablation studies convincingly demonstrate the contributions of the Transformer architecture and the alignment loss, supporting the claims of enhanced performance. The diversity in the dataset, covering various conducting styles and musical emotions, strengthens the validity of the results and showcases the model's adaptability.
While the paper provides a detailed description of the methodology and experimental setup, it lacks specific implementation details such as code availability or dataset access, which are crucial for reproducibility. The absence of a demo or project URL further limits the ability of other researchers to validate and build upon this work.
The paper acknowledges certain limitations, including the reliance on monocular reconstruction, which may not capture all nuances of conducting gestures, particularly baton motion and finger articulation. Additionally, the model struggles with very large gestures in energetic music and may lag during fast transitions. These limitations suggest areas for future research, such as incorporating hand-aware reconstruction techniques and exploring longer musical contexts.
The implications of this work extend beyond academic interest; it has potential applications in music education, virtual performances, and intelligent tutoring systems. By automating the generation of conducting gestures, this research could enhance interactive music learning environments and provide valuable tools for musicians and educators. The framework could also inspire further exploration of cross-modal motion synthesis in other artistic domains, promoting a deeper understanding of the interplay between music and movement. This paper presents a significant advancement in the field of music-driven motion synthesis through the introduction of a Transformer-based framework for generating conducting gestures. The methodology effectively combines detailed pose representation with a novel evaluation approach, setting a new standard for future research in this area.
Driven by the escalating global burden of mental health conditions, music-based interventions have attracted significant attention as a non-invasive, cost-effective modality for emotion regulation and psychological stress relief. However, current digital music services rely on static preferences and fail to adapt to users' instantaneous psychological states. Furthermore, directly mapping electroencephalography (EEG) to music generation remains challenging due to severe paired-data scarcity and a lack of interpretability. To address these limitations, we propose MindMelody, a fully functional, closed-loop real-time system for EEG-driven personalized music intervention. MindMelody introduces an emotion-mediated semantic bridge. Specifically, a hybrid Transformer-GNN first decodes real-time EEG signals into global Valence-Arousal states and local temporal affect trajectories. These states are then fed into a Retrieval-Augmented Generation (RAG)-equipped Large Language Model (LLM) to formulate structured intervention plans. Subsequently, a novel Hierarchical EEG Controller injects global affect prefixes and local temporal guidance into a pretrained music backbone, enabling fine-grained controllable audio synthesis. Crucially, the system incorporates a continuous feedback loop that updates generation parameters on the fly based on the user's evolving EEG dynamics. Extensive experiments show that MindMelody improves control adherence and emotional alignment, and receives higher perceived helpfulness in a short-term listening setting, suggesting its promise as an adaptive affect-aware music generation framework.
Primary: South China University of Technology
All Institutions: South China University of Technology
MindMelody presents a novel approach to EEG-driven personalized music intervention, demonstrating a sophisticated integration of machine learning techniques that enhance the adaptability and effectiveness of music therapy. The paper's contributions to the field of affective computing and music generation are substantial, offering a promising direction for future research and applications in mental health.
The methodology presented in MindMelody is innovative, integrating a hybrid Transformer-GNN architecture for EEG decoding with a Retrieval-Augmented Generation (RAG) mechanism to formulate structured intervention plans. The use of a Hierarchical EEG Controller to modulate a pretrained music generation backbone is particularly noteworthy, as it allows for fine-grained control over the music output based on real-time EEG data. The closed-loop feedback mechanism that continuously adapts to user feedback enhances the system's responsiveness and personalization, which is a significant advancement over static music generation systems.
The experiments conducted are robust, utilizing established datasets like DEAP for EEG affect modeling and MusicCaps for controllable music generation. The paper provides comprehensive quantitative metrics, including FAD and various subjective evaluations (Nat.-MOS, Emo.-MOS, Help.), which demonstrate the system's effectiveness in emotional alignment and perceived helpfulness. The pilot user study adds valuable qualitative insights into user experience, although it is limited in scope.
The paper includes detailed descriptions of the experimental setup, including hyperparameters and training procedures, which aids in reproducibility. However, the lack of publicly available code or a demo limits the ability for others to replicate the findings fully.
One limitation is the reliance on a relatively small dataset for training, which may affect the generalizability of the model across diverse populations. Additionally, while the pilot study shows promising results, it is not a clinical validation, and further research is needed to establish long-term efficacy and safety in real-world applications.
The potential applications of MindMelody are significant, particularly in mental health interventions, where personalized music therapy could provide non-invasive and cost-effective support for individuals experiencing emotional distress. The integration of EEG data with music generation could pave the way for more adaptive therapeutic tools in the field of affective computing. MindMelody presents a novel approach to EEG-driven personalized music intervention, demonstrating a sophisticated integration of machine learning techniques that enhance the adaptability and effectiveness of music therapy. The paper's contributions to the field of affective computing and music generation are substantial, offering a promising direction for future research and applications in mental health.
Speech technologies are deployed in high-stakes settings, yet fairness concerns remain fragmented across tasks and disciplines. Existing surveys either adopt a general machine-learning perspective that overlooks speech-specific properties or focus on a single task, missing failure patterns shared across the speech domain. Synthesizing over 400 studies spanning generation and perception tasks and emerging speech-language models, this survey presents a unified framework that links formal fairness definitions to evaluation, diagnosis, and mitigation. We formalize seven fairness definitions adapted to the speech modality and organize the field's conceptual evolution through three paradigms: Robustness, Representation, and Governance. We then ground evaluation metrics in the mathematical cores of these definitions and offer a decision tree for metric selection. We diagnose bias sources along the speech processing pipeline, surfacing speech-specific mechanisms such as channel bias as a demographic proxy and annotation subjectivity in emotion labels. We systematize mitigation strategies across four intervention stages, mapping each to the diagnosed sources. Finally, we identify open challenges and propose directions for future research.
Primary: National Taiwan University
All Institutions: National Taiwan University, University of Southern California, NTU Artificial Intelligence Center of Research Excellence
This paper serves as a foundational survey that systematically addresses bias and fairness in speech AI, providing a comprehensive framework that can guide future research and development in this critical area. The authors' approach to synthesizing existing literature and formalizing fairness definitions is a significant contribution to the field, setting the stage for more equitable speech technologies.
The paper presents a comprehensive survey that synthesizes over 400 studies related to bias and fairness in speech AI, establishing a unified framework that links formal fairness definitions to evaluation metrics, bias diagnosis, and mitigation strategies. The authors formalize seven fairness definitions specifically adapted to the speech modality and provide a decision tree for metric selection, which is a novel contribution to the field. The methodology is robust, drawing on a wide range of literature and systematically addressing the unique challenges posed by the speech domain.
While the paper is primarily a survey and does not include original experimental results, it effectively reviews existing literature and identifies gaps in current methodologies. It categorizes bias sources along the speech processing pipeline and systematizes mitigation strategies, which could serve as a foundation for future empirical studies. The depth of analysis into bias mechanisms and fairness paradigms is commendable, although the lack of original experimental validation limits the immediate applicability of the findings.
The survey does not present original experiments, thus reproducibility in the traditional sense does not apply. However, the clear organization of existing literature and the proposed frameworks allow for future researchers to build upon this work in a reproducible manner. The decision tree for metric selection is particularly useful for guiding future empirical studies.
One limitation of the paper is its reliance on existing literature without presenting new empirical data or case studies to validate the proposed frameworks. Additionally, while the survey covers a wide range of topics, it may not address all nuances of bias and fairness in speech technologies, particularly in emerging areas of research. The authors also acknowledge the complexity of navigating fairness in sociotechnical contexts, which may not be fully captured in their framework.
The implications of this work are significant, as it addresses critical issues of bias and fairness in speech technologies that are increasingly deployed in high-stakes environments. By highlighting the need for fairness as a core requirement rather than an afterthought, the paper encourages researchers and practitioners to consider the ethical implications of their technologies. This survey could influence future research directions and policy-making in the field of AI and speech technology. This paper serves as a foundational survey that systematically addresses bias and fairness in speech AI, providing a comprehensive framework that can guide future research and development in this critical area. The authors' approach to synthesizing existing literature and formalizing fairness definitions is a significant contribution to the field, setting the stage for more equitable speech technologies.
Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortions are often asymmetric, where one modality may be severely degraded while the other remains clean. Still, most contemporary AVQA metrics treat audio and video as equally reliable, causing confidence-unaware fusion to emphasize unreliable signals. This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework that explicitly estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention. The Audio-Visual Mixer utilizes frame-level, confidence-guided channel attention to gate fusion, modulating feature interaction between modalities so that high-confidence streams dominate while unreliable inputs are suppressed, preserving temporal degradation patterns. A multi-head visual confidence estimator turns frame-level artifact probabilities into temporally smoothed, clip-level visual confidence scores, while an audio confidence module derives confidence from speech-quality cues without requiring a clean reference. Experiments on multiple AVQA benchmarks show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores and yield more interpretable behavior under real-world asymmetric audio-visual distortions.
Primary: Texas State University
All Institutions: Texas State University
The paper presents MCM-AVQA, a confidence-aware audio-visual quality assessment framework that improves the robustness of quality evaluation under asymmetric distortions. This work significantly advances the state of the art in AVQA by integrating modality-specific confidence into the fusion process, leading to more accurate and interpretable quality assessments.
The proposed MCM-AVQA framework introduces a novel approach to audio-visual quality assessment by explicitly modeling modality-specific confidence and integrating it into a dedicated Audio-Visual Mixer. This methodology allows for dynamic feature gating based on confidence levels, which is a significant advancement over traditional methods that treat audio and video as equally reliable. The use of a multi-head visual confidence estimator and an audio confidence module enhances the robustness of the model under asymmetric distortions, which is a common scenario in real-world applications. The architecture is well-structured, leveraging state-of-the-art transformer models and attention mechanisms, making it a strong contribution to the field.
The experiments conducted across multiple AVQA benchmarks (LIVE-SJTU, UnB-AV, UnB-AVQ) demonstrate the effectiveness of MCM-AVQA in improving correlation with human mean opinion scores. The results indicate that the model outperforms existing state-of-the-art methods, particularly in scenarios with asymmetric distortions. The ablation studies provide valuable insights into the contributions of each component of the model, reinforcing the importance of confidence-aware fusion. The use of statistical tests to validate performance improvements adds rigor to the evaluation.
The paper provides sufficient details regarding the architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the absence of publicly available code or datasets limits the ability for other researchers to replicate the results directly. Including a project URL or demo would significantly enhance reproducibility.
One limitation of the study is the lack of a comprehensive comparison with more recent AVQA methods that may not have been included in the evaluation. Additionally, while the model shows robustness under asymmetric distortions, its performance in extreme distortion scenarios or with novel types of distortions remains untested. The reliance on subjective mean opinion scores for evaluation, while standard, could also introduce variability based on human judgment.
The MCM-AVQA framework has significant implications for real-world applications in streaming, teleconferencing, and immersive media, where audio-visual quality is critical. By improving the accuracy of quality assessments in asymmetric distortion scenarios, this work can enhance user experiences in various multimedia applications. The approach could also be extended to other multimodal quality assessment tasks, potentially influencing future research directions in the field. The paper presents MCM-AVQA, a confidence-aware audio-visual quality assessment framework that improves the robustness of quality evaluation under asymmetric distortions. This work significantly advances the state of the art in AVQA by integrating modality-specific confidence into the fusion process, leading to more accurate and interpretable quality assessments.