Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
Primary: Shanghai Innovation Institute
All Institutions: Shanghai Innovation Institute, Shanghai Jiao Tong University
GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
The paper proposes GROW, a reinforcement learning method tailored for autoregressive-diffusion Text-to-Speech (TTS) models, specifically instantiated on DiTAR. The core innovation lies in addressing the inefficiency and instability of converting deterministic ODE sampling (used in flow-matching) into Stochastic Differential Equations (SDEs) for policy gradient methods. Instead, GROW applies advantage-weighted regression directly to the flow-matching objective. It samples a group of on-policy rollouts, computes group-relative advantages for intelligibility and speaker similarity rewards, and uses these signed, normalized advantages to weight the standard flow-matching loss. A Wasserstein-2 velocity penalty is used to anchor the model to the pretrained reference, preventing divergence. The methodology is theoretically grounded, drawing on Taylor expansions to justify why linear advantage weighting outperforms exponential weighting in the high-reward, concentrated regime typical of strong pretrained TTS models. The approach is elegant in its simplicity, avoiding the overhead of trajectory storage and per-step likelihood ratio calculations required by methods like GRPO or PPO in continuous spaces.
The experimental evaluation is comprehensive and robust. The authors compare GROW against a strong baseline (DiTAR-GRPO) and the pretrained model across three benchmarks: LibriSpeech-PC, Seed-TTS EN, and Seed-TTS ZH. Results show consistent improvements in Word Error Rate (WER) and Speaker Similarity (SIM) while maintaining or slightly improving UTMOS. Notably, GROW achieves these gains with significantly higher training efficiency, being 2.9x faster than the 32-NFE DiTAR-GRPO baseline when using 10-NFE rollouts. The ablation studies are particularly strong, providing detailed analysis on the impact of rollout NFE, CFG settings, learning rates, W2 anchor strength, and reward design. The decomposition of self-imitation vs. reward-contrast provides valuable insight into why the proposed weighting scheme works better than exponential alternatives.
The paper provides extensive implementation details, including model architecture (Qwen3-0.6B backbone, LocDiT), hyperparameters (learning rates, group size, NFEs), and dataset descriptions. The authors explicitly state they will open-source the code, reproduction, and checkpoints, which significantly enhances reproducibility. The algorithm is clearly defined with pseudocode. The use of standard metrics (WER, SIM, UTMOS) and public datasets further supports reproducibility.
The method is currently evaluated primarily on zero-shot TTS tasks using specific reward models (ASR for WER, Speaker Encoder for SIM). The generalizability to other TTS objectives (e.g., emotion, prosody) or other flow-matching architectures (e.g., F5-TTS) is suggested but not extensively demonstrated. The reliance on frozen reward models introduces potential biases inherent in those models (e.g., ASR errors affecting WER rewards). Additionally, the performance gain, while consistent, is modest in absolute terms (e.g., WER reduction of ~0.45), though significant in the context of already high-performing models. The method's effectiveness on very short or very long utterances is not explicitly analyzed.
This work contributes to the efficient alignment of generative audio models, potentially lowering the computational cost of post-training for high-quality TTS systems. This could democratize access to advanced speech synthesis technologies. However, as with any TTS advancement, there are risks associated with improved voice cloning capabilities, including potential misuse for deepfakes or misinformation. The authors' open-source commitment promotes transparency and further research in this area. GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
What is music style? Though often described using text labels such as "swing," "classical," or "emotional," the real style remains implicit and hidden in concrete music examples. In this paper, we introduce a cross-modal framework that learns implicit music styles from raw audio and applies them to symbolic music generation. Inspired by BLIP-2, our model leverages a Querying Transformer (Q-Former) to extract style representations from a large, pre-trained audio language model (LM), and further applies them to condition a symbolic LM for generating piano arrangements. We adopt a two-stage training strategy: contrastive learning to align auditory style with symbolic expression, followed by generative modeling for music arrangement. Our model generates piano performances jointly conditioned on a lead sheet (content) and a reference audio example (style), enabling controllable and stylistically faithful arrangement. Experiments demonstrate the effectiveness of our approach in piano cover generation, style transfer, and audio-to-MIDI retrieval, achieving substantial improvements in style-aware alignment and music quality.
Primary: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
All Institutions: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), National University of Singapore, School of Computing
The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
The paper proposes a cross-modal framework for piano arrangement by leveraging a Querying Transformer (Q-Former) to bridge audio and symbolic domains. The core innovation lies in using the Q-Former to extract implicit style representations from a frozen audio language model (MusicGen) and conditioning a symbolic language model (MuseCoco) with these representations, alongside a lead sheet for content. The two-stage training processâcontrastive/matching learning for style alignment followed by generative modelingâis a logical extension of vision-language bootstrapping methods (like BLIP-2) to the audio-symbolic domain. The approach effectively disentangles content (lead sheet) from style (audio reference), addressing a significant gap in current music generation systems that often rely on text prompts or explicit symbolic controls. The use of LoRA adapters to condition the frozen symbolic LM is a practical and efficient design choice.
The experimental evaluation is comprehensive, covering objective metrics (MCA, CA, GPC, VCC, TA) and subjective listening tests. The inclusion of out-of-distribution datasets (Ballroom, GTZAN) strengthens the claim of generalization. The results show competitive performance against baselines like PiCoGen2 and A2M, particularly in style coherence metrics. The audio-to-MIDI retrieval task further validates the quality of the learned cross-modal representations. However, the lower content preservation scores on the in-distribution POP909 set compared to PCG2 are a notable weakness, attributed to errors in the upstream Sheetsage transcription. The subjective evaluation, while well-designed with ANOVA analysis, relies on a small sample size (21 participants), which limits the statistical power of the findings.
The paper provides detailed model configurations, including parameter counts, training epochs, batch sizes, and optimizer settings. The use of open-source or widely available backbones (MusicGen, MuseCoco, Sheetsage) enhances reproducibility. The data pairing strategy and specific metric definitions are clearly described. The lack of a public code repository is a minor drawback, but the detailed appendix and demo page mitigate this to some extent.
The authors acknowledge that the style representation is segment-level (4-bar), which may not capture long-term stylistic evolution or transitions. The reliance on Sheetsage for lead sheet extraction introduces potential error propagation. The method is currently limited to piano arrangement, and generalizing to multi-track or more complex instrumental arrangements is identified as future work. The subjective evaluation sample size is small.
This work contributes to the field of AI-assisted music creation by providing a tool for expressive piano arrangement. It enables musicians to reinterpret existing audio with specific stylistic nuances while maintaining harmonic and melodic integrity. The cross-modal representation learning technique has broader applications in audio-symbolic retrieval and style transfer. The potential for misuse in generating copyrighted music arrangements without permission is a consideration, though the focus on arrangement rather than direct cloning mitigates some risks. The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
Neural speech codecs face a fundamental tension in the language-model era: tokens that support high-fidelity reconstruction are not necessarily easy for autoregressive models to predict. Our controlled analysis of diverse codec and self-supervised speech representations shows that clearer phoneme structure before discrete code assignment is consistently associated with easier autoregressive token prediction. Yet phoneme structure alone is insufficient for high-fidelity reconstruction, which also requires reconstruction-relevant acoustic detail. Guided by this observation, we introduce ReLMCodec, a low-bitrate single-codebook speech codec built upon a preserve--control--refine principle: it preserves the linguistic organization of frozen self-supervised learning (SSL) features at the quantizer input, controls reconstruction-driven drift through Pre-quantization Anchor-Preserving Adaptation (PAPA), and refines the quantized latent space with a training-only WavLM-Large L24 teacher to reduce phoneme-level token fragmentation. Together, these components allow acoustic detail to support waveform reconstruction while keeping the resulting token sequence predictable for autoregressive models. At 650 and 800 bps, ReLMCodec moves the empirical single-stream predictability--reconstruction frontier in our evaluations, with gains that carry over to downstream text-to-speech (TTS) synthesis in both intelligibility and speaker similarity.
Primary: Peking University
All Institutions: Peking University, Tencent
[One sentence main contribution]. ReLMCodec introduces a novel single-codebook speech codec that leverages a frozen SSL anchor and residual adaptation to preserve phoneme structure, significantly improving the predictability of speech tokens for autoregressive models while maintaining high reconstruction fidelity. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by identifying and addressing the tension between reconstruction quality and token predictability in neural speech codecs. The diagnostic study providing empirical evidence linking pre-quantization phoneme structure to token predictability is a valuable insight for the community. The proposed ReLMCodec architecture, particularly the PAPA mechanism, offers a practical and effective solution to this problem. The comprehensive evaluation, including both reconstruction metrics and downstream TTS performance, strongly supports the claims. The work is well-written, clearly motivated, and technically rigorous. It represents a solid advancement in the field of speech tokenization for LLM-based generation.
The paper proposes ReLMCodec, a single-codebook speech codec designed to optimize the trade-off between waveform reconstruction quality and autoregressive token predictability. The core methodological contribution is the "preserve--control--refine" principle. It preserves linguistic structure by anchoring the quantizer input to frozen W2v-BERT 2.0 features. It controls reconstruction drift via Pre-quantization Anchor-Preserving Adaptation (PAPA), which adds a scaled residual correction to the anchor rather than learning the full representation. It refines the latent space using a training-only WavLM-Large teacher to reduce phoneme-level fragmentation. The approach is technically sound and addresses a specific, well-defined problem in the intersection of speech coding and LLM-based generation. The use of a frozen SSL anchor to maintain phoneme structure while allowing acoustic detail to be learned via a residual path is a clever and effective architectural choice.
The experimental evaluation is comprehensive. The authors conduct a controlled diagnostic study across 24 different codec and SSL representations, establishing a strong correlation between pre-quantization phoneme structure (measured by KNN accuracy) and token predictability (P-ACC/P-PPL). This diagnostic provides a strong theoretical grounding for the design choices. The end-to-end results on LibriSpeech show that ReLMCodec achieves state-of-the-art performance at low bitrates (650/800 bps) in terms of WER, PESQ, and speaker similarity compared to baselines like X-Codec2, FocalCodec, and SemantiCodec. The downstream TTS evaluation further validates the utility of the tokens, showing improved intelligibility and speaker similarity. The ablation studies effectively isolate the contributions of PAPA and the JMAS refinement loss.
The paper provides detailed descriptions of the architecture, training objectives, and evaluation protocols. The appendix includes extensive details on the probing protocol, baseline configurations, and statistical methods. The code release statement indicates that checkpoints and scripts will be available upon acceptance. The use of standard datasets (LibriSpeech) and metrics (WER, PESQ, UTMOS) enhances reproducibility. The matched probing protocol is particularly well-documented, allowing other researchers to replicate the diagnostic analysis.
The study is limited to English speech (LibriSpeech). The analysis of phoneme structure relies on forced alignment (MFA), which may not be robust to noisy or out-of-domain data. The teacher selection is limited to W2v-BERT and WavLM; the generalizability to other SSL models is not explored. The paper acknowledges that multilingual and noisy settings are future work. Additionally, the reliance on a training-only teacher adds complexity to the training pipeline, although it does not affect inference cost.
ReLMCodec contributes to the development of efficient and predictable speech tokenizers, which are critical for the advancement of end-to-end speech generation models based on LLMs. By improving the predictability of tokens, it may facilitate the training of smaller, more efficient speech LLMs. The insights gained from the diagnostic study could inform the design of future speech representations. The work has positive implications for the field of audio AI, promoting more efficient and linguistically aware speech processing. [One sentence main contribution]. ReLMCodec introduces a novel single-codebook speech codec that leverages a frozen SSL anchor and residual adaptation to preserve phoneme structure, significantly improving the predictability of speech tokens for autoregressive models while maintaining high reconstruction fidelity. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by identifying and addressing the tension between reconstruction quality and token predictability in neural speech codecs. The diagnostic study providing empirical evidence linking pre-quantization phoneme structure to token predictability is a valuable insight for the community. The proposed ReLMCodec architecture, particularly the PAPA mechanism, offers a practical and effective solution to this problem. The comprehensive evaluation, including both reconstruction metrics and downstream TTS performance, strongly supports the claims. The work is well-written, clearly motivated, and technically rigorous. It represents a solid advancement in the field of speech tokenization for LLM-based generation.
Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models. Project page and release updates are available at https://github.com/ryysayhi/AudioMap.
Primary: Kling Team
All Institutions: Kling Team
AudioMap introduces a novel RL-based framework for time-aware dense audio captioning, utilizing a cloze-and-choice reward paradigm to enhance fine-grained semantic coverage and temporal grounding, achieving state-of-the-art results among open-source models.
The paper proposes AudioMap, an RL-based framework for Time-Aware Dense Audio Captioning (TDAC). The core methodological contribution is a shift from coarse scalar rewards to a "cloze-and-choice" paradigm. Specifically, it introduces the Evidence Sufficiency Reward (ESR), which uses a frozen examiner to generate multiple-choice questions about acoustic details, and the Event-Conditioned Temporal Reward (ECTR), which aligns event descriptions with timestamps using temporal IoU. The approach employs Group Relative Policy Optimization (GRPO) with a dual-curriculum strategy (input duration and reward complexity). The methodology is technically sound and addresses specific pain points in dense captioning (lack of fine-grained supervision and temporal grounding). However, the use of cloze tests for reward modeling is not entirely new in the broader LLM literature (e.g., Omni-Cloze), though its application to dense audio captioning with temporal constraints is a novel adaptation. The reliance on a large frozen examiner (Qwen3.6-27B) for reward computation is a significant computational overhead.
The authors construct a new dataset, AudioMapCap-44K, with 44K high-quality captions. Experiments show SOTA performance among open-source models on Omni-Cloze, MMSU, MMAU, and TACOS benchmarks. The results are competitive with proprietary models like Gemini-3.1-Pro. The ablation studies effectively demonstrate the contribution of ESR (semantic coverage) and ECTR (temporal grounding). The user study provides subjective validation. The evaluation is comprehensive, covering semantic, QA-based, and temporal metrics. The comparison with proprietary models is strong.
The paper provides a GitHub link. The methodology describes the training setup (Qwen2.5-Omni initialization, GRPO details, curriculum stages). However, the reliance on proprietary models for dataset construction (Gemini-3.1-Pro, GPT-4.1) and the specific prompt engineering for the cloze question generation might make exact replication difficult without the specific prompts or access to those models. The code release is a positive sign.
The method is computationally expensive due to the need for a large examiner model during RL training. The dataset construction relies heavily on proprietary LLMs, which may introduce biases or limitations in the diversity of the data. The temporal grounding is still dependent on the examiner's ability to extract timestamps, which can be error-prone for concurrent events. The paper does not extensively discuss the failure cases of the temporal reward in complex acoustic scenes.
This work advances the field of audio understanding by enabling more detailed and temporally precise audio descriptions, which is crucial for accessibility, multimedia retrieval, and embodied AI. The dataset AudioMapCap-44K is a valuable resource for the community. The RL-based approach offers a generalizable framework for other dense captioning tasks. AudioMap introduces a novel RL-based framework for time-aware dense audio captioning, utilizing a cloze-and-choice reward paradigm to enhance fine-grained semantic coverage and temporal grounding, achieving state-of-the-art results among open-source models.
Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.
Primary: Zhejiang University
All Institutions: Zhejiang University, The Chinese University of Hong Kong, Shandong University, Ant Group
MusicLayout introduces an explicit, time-aligned intermediate representation for controlling musical structure in text-to-music generation, enabling interpretable planning and layout-level control within a unified autoregressive framework.
The paper proposes MusicLayout, an explicit intermediate representation for text-to-music generation. The core methodological contribution is the design of a structured, time-aligned token sequence that describes musical sections, textures, variations, and instrument arrangements. This layout is generated by a unified autoregressive language model (based on ACE-Step 1.5) before audio tokens are predicted. The approach effectively introduces a "planning" layer, analogous to spatial layout planning in image generation, but adapted for the temporal domain of music. The serialization grammar is well-defined, and the integration into the autoregressive sequence is straightforward. However, the novelty is somewhat incremental; it applies a known paradigm (explicit intermediate representation) to a new modality (music) without introducing fundamentally new algorithmic mechanisms for the generation process itself. The reliance on MIDI-synthesized audio for training the layout structure is a significant methodological constraint, as it decouples the structural planning from the acoustic fidelity of the final output.
The evaluation is comprehensive, covering objective metrics (FAD, CLAPScore, PaSST-KL, SSIM, SCM Energy Distance, boundary agreement) and subjective human evaluation. The authors include strong baselines (MusicGen, ACE-Step, Stable Audio) and, crucially, matched-data controls (finetuned ACE-Step without layout, shuffled layout training/inference) to isolate the effect of the explicit planning. The results show that MusicLayout improves structural organization (boundary agreement) and can be manipulated to control structure. However, the acoustic quality metrics (FAD, CLAP) are generally inferior to or comparable with baselines, which the authors attribute to the use of MIDI-synthesized training data. The evaluation on MuChin (real audio) shows a performance drop, highlighting the domain gap. The subjective evaluation supports the claim of improved structural control and text consistency for experienced listeners. The ablation studies are rigorous and help validate the specific contribution of the layout tokens.
The paper provides detailed implementation details, including the model architecture (ACE-Step 1.5 backbone), training settings (batch size, learning rate, GPU count), and the specific datasets used (FreeMIDI, MidiCaps, MuChin). The annotation pipeline for creating MusicLayout from MIDI is described in an algorithmic format. The code for the layout extraction and the model adaptation is not explicitly linked, but the description is sufficient for a competent researcher to reproduce the method. The use of open-source components (FluidSynth, PrettyMIDI, ACE-Step) aids reproducibility.
The primary limitation is the reliance on MIDI-synthesized audio for training the structural planner. This creates a domain gap when evaluating on real audio (MuChin), where performance degrades. The system cannot edit existing audio or regenerate selected regions; it only allows pre-synthesis layout manipulation. The vocabulary for sections, textures, and instruments is closed and may not cover all musical styles or nuances. The quality of the generated layout depends on the text prompt, and if the prompt is ambiguous, the generated layout may not align with the user's intent. The acoustic fidelity is limited by the quality of the MIDI synthesis used in training, although the DiT renderer is frozen and presumably capable of higher quality if trained on better data.
This work contributes to the field of controllable generative AI by demonstrating the value of explicit intermediate representations for complex, structured data like music. It opens up new avenues for human-in-the-loop music creation, where users can plan the structure of a piece before generation. This could lower the barrier to entry for music production and enable more precise artistic control. However, the reliance on MIDI data and the potential for generating low-quality audio from poor MIDI synthesis are practical concerns. The work also raises questions about the generalizability of such explicit planning methods to other domains. MusicLayout introduces an explicit, time-aligned intermediate representation for controlling musical structure in text-to-music generation, enabling interpretable planning and layout-level control within a unified autoregressive framework.
Many music datasets contain MIDI notes but lack reliable velocities, defaulting to a constant value. This absence is especially problematic outside the piano domain, as velocity is a core component for expressive rendering, music generation, and performance analysis. This paper studies cross-instrument MIDI velocity estimation in this label-scarce setting. Starting from a piano-trained velocity estimator, we recast target-instrument adaptation as predicting renderer-conditioned velocities whose rendering matches the dynamics of the performance audio. This adaptation can be driven by either differentiable synthesizers (Diff-Synth) or our proposed differentiable SoundFont proxies (Diff-SFProxy). We highlight the Diff-SFProxy: it supervises velocity through note-wise, loudness-related acoustic parameters rather than waveform reconstruction, focusing gradients on velocity-dependent behavior. Experiments on piano and guitar show that Diff-SFProxy is effective for cross-instrument MIDI velocity estimation, while waveform-domain Diff-Synth degrades performance.
Primary: University of Western Australia
All Institutions: University of New South Wales, University of Western Australia
The paper presents a robust and innovative solution to cross-instrument MIDI velocity estimation by introducing a differentiable SoundFont proxy that leverages perceptual loudness parameters for gradient-based adaptation, effectively overcoming the limitations of waveform-based methods in label-scarce settings.
The paper proposes a novel framework for cross-instrument MIDI velocity estimation in label-scarce settings. The core innovation is the "Diff-SFProxy," a differentiable proxy that maps MIDI note events to loudness-related acoustic parameters (Pitch-conditioned Harmonic Energy and Onset-window Spectral Flux) derived from SoundFont renders. This allows gradient-based adaptation of a piano-trained velocity estimator (VeloEst) to target instruments (guitar) without requiring ground-truth velocity labels. The methodology effectively addresses the "gradient mismatch" problem inherent in waveform-based differentiable synthesis (Diff-Synth), where timbral and room-acoustic mismatches dominate the loss landscape. The use of perceptual loudness metrics (Bark-scale specific/total loudness) for evaluation is a strong methodological choice, aligning the objective with the perceptual goal of velocity estimation.
The experimental design is rigorous and well-structured. The authors include a critical "velocity-recovery diagnostic" on synthetic audio to verify that the Diff-SFProxy gradients point in the correct direction, which is essential for trusting the adaptation process. The evaluation on real-world datasets (MAESTRO, SMD, GAPS, FL) demonstrates that Diff-SFProxy significantly outperforms both flat-velocity baselines and zero-shot transfer. Crucially, it shows that Diff-Synth often degrades performance on guitar, validating the paper's central hypothesis that parameter-space supervision is superior to waveform-space supervision for this specific task. The ablation studies on segment length and loss components further strengthen the findings.
The paper provides substantial implementation details, including FFT settings, optimizer hyperparameters, and architecture specifications for the Transformer proxy. The code is explicitly linked via GitHub. The use of standard datasets (MAESTRO, GAPS) and public SoundFonts enhances reproducibility. The description of the training protocol, including the mixture sampler and caching strategy, is clear enough for replication.
The approach relies on the availability of a suitable SoundFont for the target instrument, which requires manual selection or future automation. The recovered velocities are "renderer-conditioned," meaning they are optimal for the specific SoundFont used, not necessarily the canonical physical velocity of the performance. The method is currently limited to instruments with available SoundFonts and may struggle with complex polyphony or instruments with significant inharmonicity not captured by the proxy. The evaluation is limited to piano and guitar; generalization to other instruments (violin, wind) is speculative.
This work addresses a significant bottleneck in symbolic music processing: the lack of expressive velocity data for non-piano instruments. By enabling label-scarce adaptation, it facilitates the creation of more expressive MIDI representations for music generation, analysis, and education. The Diff-SFProxy framework could be extended to other expressive parameters (e.g., pedal, bow pressure), potentially enriching the symbolic music domain. The paper presents a robust and innovative solution to cross-instrument MIDI velocity estimation by introducing a differentiable SoundFont proxy that leverages perceptual loudness parameters for gradient-based adaptation, effectively overcoming the limitations of waveform-based methods in label-scarce settings.
Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which speech and background audio are independently manipulated over otherwise authentic video. Yet existing research either focuses on visual manipulation, addresses speech detection in isolation, or conflates speech and non-speech audio as a single undifferentiated audio stream, overlooking the distinct forensic challenges posed by background audio. This conflation is consequential: the two acoustic components arise from fundamentally different generative mechanisms, exhibit distinct artifact profiles, and pose different challenges to detection systems. We introduce MADBench, the first benchmark that treats speech and environmental audio as distinct acoustic components, enabling component-aware evaluation of audio deepfake detection across independently manipulated forgery sources. We benchmark representative state-of-the-art detectors and multimodal large language models under a unified protocol. Our experiments reveal that environmental audio manipulation is more detectable than synthetic speech across general-purpose encoders, while existing pretrained detectors fail on both acoustic components, and manipulated environmental audio asymmetrically degrades speech deepfake detection, findings entirely invisible under the single-label paradigm of prior benchmarks. MADBench establishes a rigorous foundation for future research into robust, component-aware audio deepfake detection.
Primary: Unknown
All Institutions: Unknown
The paper introduces MADBench, a novel benchmark for component-aware audio deepfake detection that disentangles speech and environmental audio manipulation, providing rigorous evaluation protocols and insightful findings on detector behavior and cross-component interference.
The paper proposes MADBench, a novel benchmark for audio deepfake detection that disentangles speech and environmental audio components. The methodology involves a rigorous pipeline: source video separation using MossFormer2, generation of fake speech via TTS/VC models (F5-TTS, SeedVC, etc.), and generation of fake environmental audio via Text-to-Audio and Video-to-Audio models (AudioLDM2, MMAudio, etc.). A key methodological contribution is the introduction of a "scene-consistency" axis (matched vs. mismatched environmental audio) and the evaluation of cross-component interference. The approach is technically sound, addressing a genuine gap in existing benchmarks which often conflate audio modalities or focus solely on speech. However, the novelty is somewhat tempered by the fact that the core components (separation, TTS, TTA) are established techniques; the innovation lies primarily in the specific combination and the evaluation protocol rather than a new algorithmic breakthrough.
The experimental evaluation is comprehensive. The authors benchmark a wide range of models: pretrained AV detectors (AVH-Align, etc.), frozen AV encoders (ImageBind, PE-AV, CAV-MAE), and zero-shot Omni models (Qwen2.5-Omni, MiniCPM-o). They evaluate across multiple protocols (Binary, 4-way, Component-level, Scene-consistency). The results are insightful: they find that environmental audio manipulation is easier to detect than speech, and that fake environmental audio can obscure speech detection (cross-component interference). They also find that frozen AV encoders perform surprisingly well, often better than task-specific detectors, and that video input does not consistently improve manipulation detection but helps with scene consistency. The analysis is thorough, including ablation studies on input modality and scene consistency. The statistical reporting (AUC, EER) is appropriate.
The paper provides detailed descriptions of the dataset construction, including the source dataset (AVSpeech), the separation model (MossFormer2), the specific generation models used, and the quality control steps. The split strategy (7:1:2) and the balancing of generation settings are described. However, the paper does not provide a public link to the dataset or code in the text provided (URLs are "none"). The use of proprietary or hard-to-access models (e.g., specific checkpoints of F5-TTS, SeedVC, etc.) might pose reproducibility challenges for the exact generation pipeline, although the models themselves are generally available. The detailed protocol allows for reasonable reproducibility of the evaluation framework.
The paper acknowledges that the visual stream is held fixed, which limits the study to audio manipulation detection in authentic video. It does not address scenarios where both audio and video are manipulated. The dataset size (1,892 clips) is relatively small for deep learning benchmarks, which might limit the generalizability of the findings, especially for complex multimodal models. The reliance on automatic scene classification (Qwen2.5-VL) for taxonomy construction introduces potential noise, although manual verification was performed. The performance of zero-shot models is limited, which is expected but highlights the current gap in multimodal reasoning for forensic tasks.
This paper has significant broader impact. By providing a benchmark that reflects a realistic attack scenario (independent manipulation of speech and background), it enables more robust development of deepfake detection systems. The findings that environmental audio manipulation is easier to detect and can interfere with speech detection have practical implications for security and forensics. The benchmark will likely spur research into more robust, component-aware detection models. It also highlights the limitations of current zero-shot multimodal models in forensic reasoning, guiding future research directions. The paper introduces MADBench, a novel benchmark for component-aware audio deepfake detection that disentangles speech and environmental audio manipulation, providing rigorous evaluation protocols and insightful findings on detector behavior and cross-component interference.
Neural speech codecs face a fundamental tension in the language-model era: tokens that support high-fidelity reconstruction are not necessarily easy for autoregressive models to predict. Our controlled analysis of diverse codec and self-supervised speech representations shows that clearer phoneme structure before discrete code assignment is consistently associated with easier autoregressive token prediction. Yet phoneme structure alone is insufficient for high-fidelity reconstruction, which also requires reconstruction-relevant acoustic detail. Guided by this observation, we introduce ReLMCodec, a low-bitrate single-codebook speech codec built upon a preserve--control--refine principle: it preserves the linguistic organization of frozen self-supervised learning (SSL) features at the quantizer input, controls reconstruction-driven drift through Pre-quantization Anchor-Preserving Adaptation (PAPA), and refines the quantized latent space with a training-only WavLM-Large L24 teacher to reduce phoneme-level token fragmentation. Together, these components allow acoustic detail to support waveform reconstruction while keeping the resulting token sequence predictable for autoregressive models. At 650 and 800 bps, ReLMCodec moves the empirical single-stream predictability--reconstruction frontier in our evaluations, with gains that carry over to downstream text-to-speech (TTS) synthesis in both intelligibility and speaker similarity.
Primary: Peking University
All Institutions: Peking University, Tencent
[One sentence main contribution]. ReLMCodec introduces a novel single-codebook speech codec that leverages a frozen SSL anchor and residual adaptation to preserve phoneme structure, significantly improving the predictability of speech tokens for autoregressive models while maintaining high reconstruction fidelity. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by identifying and addressing the tension between reconstruction quality and token predictability in neural speech codecs. The diagnostic study providing empirical evidence linking pre-quantization phoneme structure to token predictability is a valuable insight for the community. The proposed ReLMCodec architecture, particularly the PAPA mechanism, offers a practical and effective solution to this problem. The comprehensive evaluation, including both reconstruction metrics and downstream TTS performance, strongly supports the claims. The work is well-written, clearly motivated, and technically rigorous. It represents a solid advancement in the field of speech tokenization for LLM-based generation.
The paper proposes ReLMCodec, a single-codebook speech codec designed to optimize the trade-off between waveform reconstruction quality and autoregressive token predictability. The core methodological contribution is the "preserve--control--refine" principle. It preserves linguistic structure by anchoring the quantizer input to frozen W2v-BERT 2.0 features. It controls reconstruction drift via Pre-quantization Anchor-Preserving Adaptation (PAPA), which adds a scaled residual correction to the anchor rather than learning the full representation. It refines the latent space using a training-only WavLM-Large teacher to reduce phoneme-level fragmentation. The approach is technically sound and addresses a specific, well-defined problem in the intersection of speech coding and LLM-based generation. The use of a frozen SSL anchor to maintain phoneme structure while allowing acoustic detail to be learned via a residual path is a clever and effective architectural choice.
The experimental evaluation is comprehensive. The authors conduct a controlled diagnostic study across 24 different codec and SSL representations, establishing a strong correlation between pre-quantization phoneme structure (measured by KNN accuracy) and token predictability (P-ACC/P-PPL). This diagnostic provides a strong theoretical grounding for the design choices. The end-to-end results on LibriSpeech show that ReLMCodec achieves state-of-the-art performance at low bitrates (650/800 bps) in terms of WER, PESQ, and speaker similarity compared to baselines like X-Codec2, FocalCodec, and SemantiCodec. The downstream TTS evaluation further validates the utility of the tokens, showing improved intelligibility and speaker similarity. The ablation studies effectively isolate the contributions of PAPA and the JMAS refinement loss.
The paper provides detailed descriptions of the architecture, training objectives, and evaluation protocols. The appendix includes extensive details on the probing protocol, baseline configurations, and statistical methods. The code release statement indicates that checkpoints and scripts will be available upon acceptance. The use of standard datasets (LibriSpeech) and metrics (WER, PESQ, UTMOS) enhances reproducibility. The matched probing protocol is particularly well-documented, allowing other researchers to replicate the diagnostic analysis.
The study is limited to English speech (LibriSpeech). The analysis of phoneme structure relies on forced alignment (MFA), which may not be robust to noisy or out-of-domain data. The teacher selection is limited to W2v-BERT and WavLM; the generalizability to other SSL models is not explored. The paper acknowledges that multilingual and noisy settings are future work. Additionally, the reliance on a training-only teacher adds complexity to the training pipeline, although it does not affect inference cost.
ReLMCodec contributes to the development of efficient and predictable speech tokenizers, which are critical for the advancement of end-to-end speech generation models based on LLMs. By improving the predictability of tokens, it may facilitate the training of smaller, more efficient speech LLMs. The insights gained from the diagnostic study could inform the design of future speech representations. The work has positive implications for the field of audio AI, promoting more efficient and linguistically aware speech processing. [One sentence main contribution]. ReLMCodec introduces a novel single-codebook speech codec that leverages a frozen SSL anchor and residual adaptation to preserve phoneme structure, significantly improving the predictability of speech tokens for autoregressive models while maintaining high reconstruction fidelity. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper makes a significant technical contribution by identifying and addressing the tension between reconstruction quality and token predictability in neural speech codecs. The diagnostic study providing empirical evidence linking pre-quantization phoneme structure to token predictability is a valuable insight for the community. The proposed ReLMCodec architecture, particularly the PAPA mechanism, offers a practical and effective solution to this problem. The comprehensive evaluation, including both reconstruction metrics and downstream TTS performance, strongly supports the claims. The work is well-written, clearly motivated, and technically rigorous. It represents a solid advancement in the field of speech tokenization for LLM-based generation.
India's linguistic landscape spans over 700 languages and thousands of dialects, yet the vast majority of automatic speech recognition (ASR) systems support only a small fraction of this diversity. We present SraVaani-1.0, a multilingual ASR model covering 65 Indian languages and dialects, many of which currently have no publicly available or competing ASR system. SraVaani-1.0 is built on a FastConformer architecture and trained from scratch through a three-stage pipeline.In the first stage, we perform self-supervised pretraining on 31,255 hours of unlabelled speech from the VAANI corpus using a contrastive learning objective. In the second stage, we introduce an audio-image representation alignment stage that leverages the paired images and speech available in the VAANI corpus. This multimodal alignment encourages the speech encoder to learn semantically richer representations by exploiting the relationship between visual context and spoken content, thereby improving downstream recognition, particularly for low-resource languages.In the final stage, the aligned encoder is fine-tuned end-to-end using a Hybrid Token-and-Duration Transducer (TDT)-CTC decoder on 31,263 hours of labelled multilingual Indian speech compiled from 24 public datasets spanning 65 languages and dialects. We evaluate SraVaani-1.0 against three state-of-the-art multilingual ASR systems across eight benchmarks. SraVaani-1.0 achieves the lowest word error rate (WER) on a large number of language-dataset pairs while remaining competitive with the best-performing systems on high-resource languages.Most importantly, it is the only open-source evaluated model that provides transcription capability for multiple low-resource and tribal Indian languages, which are assessed exclusively on the VAANI benchmark.
Primary: Indian Institute of Science (IISc)
All Institutions: Indian Institute of Science (IISc), SPIRE Lab, ARTPARK@IISc
SraVaani-1.0 is a significant contribution to the field of multilingual speech recognition, particularly for low-resource Indian languages, by demonstrating the effectiveness of a three-stage pipeline combining self-supervised learning, audio-image alignment, and supervised fine-tuning, and by providing the first open-source ASR system with broad coverage of India's linguistic diversity.
The paper proposes a three-stage training pipeline for a multilingual ASR system. Stage 1 involves self-supervised contrastive pretraining on a large corpus of unlabelled Indian speech. Stage 2 introduces a novel audio-image representation alignment stage, leveraging paired audio and images from the VAANI corpus to inject semantic context into the speech encoder using a SigLIP-style contrastive loss. Stage 3 is supervised fine-tuning using a Hybrid Token-and-Duration Transducer (TDT)-CTC decoder. The methodology is technically sound and follows established practices in self-supervised learning (wav2vec 2.0 style) and multimodal alignment (SigLIP). The specific novelty lies in the application of audio-image alignment to improve low-resource ASR performance, which is a creative use of available multimodal data. However, the core components (FastConformer, TDT, SigLIP) are not new; the contribution is primarily in the system design and data curation for a specific linguistic domain.
The evaluation is comprehensive, covering 65 languages and comparing against strong baselines (Gemini 3 Flash, Sarvam Saaras v3, IndicConformer). The use of multiple benchmarks (CommonVoice, FLEURS, etc.) provides a robust assessment of generalization. The results demonstrate that SraVaani-1.0 achieves state-of-the-art performance on many languages, particularly those with limited prior support. The inclusion of 44 languages with no competing ASR systems is a significant empirical contribution. The analysis of the coverage-accuracy trade-off is insightful. However, the WERs on some low-resource languages are quite high (e.g., Nyishi 93.9%), indicating that while the model is the best available, it is not yet production-ready for all target languages. The evaluation is limited to in-domain data (VAANI), which the authors acknowledge as a limitation.
The paper provides detailed implementation details, including model architecture, hyperparameters, training schedules, and data preprocessing steps. The use of standard frameworks (NVIDIA NeMo, PyTorch Lightning) and public datasets enhances reproducibility. The model weights are made available on Hugging Face, which is a strong positive for reproducibility and community impact. The codebase is not explicitly mentioned as open-source, but the model weights are.
The authors openly discuss several limitations: lack of support for Urdu and Kashmiri in fine-tuning, issues with code-switched speech transcription, lack of text normalization, unreliable language identification, and significant variance in data availability across languages. The evaluation is also limited to in-domain data, which may not reflect real-world performance on out-of-domain speech. The high WERs on very low-resource languages highlight the current limits of the approach.
This work has significant potential impact on digital equity in India by providing ASR capabilities for a vast number of underrepresented languages and dialects. This can facilitate access to government services, education, and healthcare for rural and tribal communities. The open-source release of the model promotes further research and development in low-resource NLP and speech technologies. The approach of leveraging multimodal data for ASR improvement could be applicable to other domains and languages. SraVaani-1.0 is a significant contribution to the field of multilingual speech recognition, particularly for low-resource Indian languages, by demonstrating the effectiveness of a three-stage pipeline combining self-supervised learning, audio-image alignment, and supervised fine-tuning, and by providing the first open-source ASR system with broad coverage of India's linguistic diversity.
LLM-based full-duplex voice services allow users to speak while the assistant is responding. Because servers can generate output and advance dialogue state faster than clients can play it, subsequent user speech may be interpreted based on content the user never heard. We call this failure Generative Context Mis-anchoring (GCM). To address GCM issues, we present PACE, a provider-independent middleware layer that anchors model-facing context to the client playback boundary, a system-observable proxy for what the user could have heard. After an interruption, PACE repairs this context to exclude assistant content that never reached playback, while preserving low-latency generation across heterogeneous voice runtimes. We implement PACE's audio-only projection path end to end in a browser-based realtime voice assistant using a black-box speech model, without modifying the model service. We also construct GCM-Bench, a new controlled benchmark dataset of 108 playback-relative referent-anchoring cases. On GCM-Bench, PACE raises Referent Anchoring Accuracy from 25.0% to 96.3% over a cancellation-only baseline. On 200 Full-Duplex-Bench v1 interruption samples, it preserves interruption response quality. These results show that grounding model-facing context in actual playback is a practical way to maintain consistency in full-duplex voice dialogue.
Primary: Alibaba Group
All Institutions: Alibaba Group
The paper presents a practical systems solution for context misalignment in full-duplex voice LLMs, demonstrating significant accuracy improvements on a new benchmark through playback-aware context repair.
The paper introduces PACE, a middleware layer designed to solve "Generative Context Mis-anchoring" (GCM) in full-duplex voice LLM systems. The core innovation is decoupling the model's internal dialogue state from the client's actual playback boundary. By tracking audio samples rendered to the user, PACE identifies which assistant content was actually heard. Upon interruption, it "repairs" the context by revoking unplayed content before passing the user's input to the model. This is a systems-level architectural contribution rather than a novel algorithmic or modeling breakthrough. The approach is pragmatic, leveraging existing APIs (like OpenAI's item truncation) but providing a generalized abstraction for state management across heterogeneous runtimes. The methodology is sound for its specific problem domain but relies on standard software engineering patterns (state tracking, event handling) rather than new machine learning techniques.
The authors evaluate PACE using two benchmarks. First, GCM-Bench, a new dataset of 108 cases specifically designed to test referent anchoring accuracy. PACE improves accuracy from 25.0% to 96.3% compared to a baseline that only cancels generation without context repair. This is a significant and clear improvement, demonstrating the necessity of context repair. Second, they evaluate on 200 samples from Full-Duplex-Bench v1, showing that PACE preserves interruption response quality (mean score 4.975 vs 4.995). The evaluation is relevant and directly addresses the proposed problem. However, the GCM-Bench is small (108 cases) and likely synthetic or narrowly curated, which limits the generalizability of the accuracy claims. The latency impact is mentioned but not deeply quantified in terms of added overhead, which is critical for real-time voice systems.
The paper provides a link to a GitHub repository for the GCM-Bench dataset. The implementation is described as a browser-based middleware. While the code for the benchmark is available, the full source code for the PACE middleware itself is not explicitly linked in the provided text (only the benchmark repo is cited). The description of the "audio-only projection path" is detailed enough for a systems researcher to replicate the logic, but the exact integration with specific black-box models might require additional inference. The use of a standardized benchmark (FDB v1) aids in reproducibility of the interruption quality metrics.
The primary limitation is the scope of the evaluation. GCM-Bench is small and potentially lacks the diversity of natural conversation nuances found in larger, open-domain datasets. The paper focuses on a "black-box" projection path; while this is a strength for portability, it may not achieve the same precision as a system with deep access to internal model states (like KV-cache truncation), which the paper acknowledges but does not benchmark against in terms of ultimate accuracy ceiling. Additionally, the paper does not extensively discuss the computational overhead of the middleware itself, which could be non-trivial in high-concurrency scenarios. The assumption that playback boundary is a perfect proxy for "what the user heard" ignores user attention and cognitive processing, though the authors correctly note this distinction.
This work addresses a critical usability issue in the deployment of full-duplex voice assistants. As LLM-based voice interfaces become more common, ensuring that the system's context aligns with the user's actual experience is vital for trust and effectiveness. PACE provides a practical, provider-independent solution that can be adopted by various voice service providers. It highlights the importance of systems-level design in multimodal AI, showing that even with powerful models, poor state management can lead to significant failures. This could influence the design of future voice assistant frameworks to prioritize playback-aware context management. The paper presents a practical systems solution for context misalignment in full-duplex voice LLMs, demonstrating significant accuracy improvements on a new benchmark through playback-aware context repair.
Continuous-latent autoregressive speech generation has emerged as a promising alternative to discrete-token modeling by avoiding quantization loss and preserving richer acoustic information. However, continuous acoustic targets do not ex- pose linguistic structure as explicit token-level prediction tar- gets. Consequently, the autoregressive language model (LM) must acquire linguistic structure indirectly through acous- tic prediction, which can compromise the content fidelity of generated speech. We propose SemBridge, a training-only semantic-token anchoring framework for continuous-latent autoregressive speech generation. SemBridge uses discrete se- mantic tokens to directly supervise autoregressive LM states and employs a Semantic-Aligned Acoustic VAE to organize the continuous target space under the same semantic refer- ence. The semantic supervision is used only during train- ing, while inference remains entirely continuous. We evalu- ate SemBridge on zero-shot text-to-speech (TTS) and score- conditioned singing voice synthesis (SVS). Across multi- ple benchmarks, SemBridge improves content accuracy, as measured by word and character error rates (WER/CER), while maintaining competitive speaker similarity and percep- tual quality. Experimental results demonstrate that explicit semantic-token supervision for autoregressive state learning is an effective and general direction for continuous speech generation. Speech samples are available.1 The model code and checkpoints will be available at https://github.com/ASLP- lab/SemBridge
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Soul AI Lab
SemBridge introduces a training-only semantic-token anchoring framework that improves content fidelity in continuous-latent autoregressive speech generation by supervising LM states with discrete semantic tokens, demonstrating significant gains in WER/CER across TTS and SVS benchmarks while maintaining competitive acoustic quality.
The paper proposes SemBridge, a framework for continuous-latent autoregressive speech generation. The core innovation is "semantic-token anchoring," which uses discrete semantic tokens (from a frozen GLM-4 tokenizer) to supervise the hidden states of the autoregressive language model during training. This is complemented by a Semantic-Aligned Acoustic VAE (SA-VAE) that aligns continuous acoustic latents with semantic embeddings. The approach is technically sound and addresses a real gap in continuous speech generation: the lack of explicit linguistic structure in continuous targets. However, the novelty is moderate; using discrete tokens to guide continuous generation is a known strategy (e.g., in VALL-E X, SpeechTokenizer), and the specific implementation of anchoring LM states with classification loss is a straightforward adaptation of existing multi-task learning principles. The method is essentially a training-time regularization technique that does not alter the inference architecture, which is a pragmatic but not groundbreaking design choice.
The evaluation is comprehensive, covering zero-shot TTS on Seed-TTS-Eval and CV3-Eval, as well as score-conditioned SVS on GMO-SVS. The results show consistent improvements in content accuracy (WER/CER) compared to baselines like MELA-TTS and VoxCPM, while maintaining competitive speaker similarity and perceptual quality. The ablation studies are well-conducted, isolating the effects of target-space alignment, state anchoring, anchoring depth, and anchoring strength. The inclusion of SVS results adds value by demonstrating the method's transferability to a more challenging task with explicit musical constraints. The comparison with a wide range of baselines (discrete, continuous, and hybrid) provides a strong context for the results. The paper effectively demonstrates that explicit semantic supervision improves content fidelity without sacrificing acoustic quality.
The paper provides detailed descriptions of the model architecture, training stages, hyperparameters, and datasets. It specifies the use of the GLM-4 tokenizer and the SA-VAE architecture. The training configurations, including batch sizes, learning rates, and loss weights, are clearly listed. The authors state that code and checkpoints will be available, and a demo link is provided. The detailed ablation studies and clear separation of Stage I and Stage II training procedures enhance reproducibility. However, the reliance on specific internal datasets (Soul AI Lab's 20K-hour singing data) for the joint TTS-SVS model limits full reproducibility for that specific configuration, though the TTS-only experiments are likely reproducible with the public VoxBox corpus.
The primary limitation is that the semantic supervision is only applied during training. While this keeps inference efficient, it means the model relies entirely on the learned alignment between continuous latents and discrete tokens, which might not be as robust as end-to-end discrete modeling in some edge cases. The paper notes a trade-off between content accuracy and synthesis quality depending on the anchoring depth and weight, suggesting that hyperparameter tuning is critical. Additionally, the use of a frozen semantic tokenizer means the model cannot adapt the semantic representation to the specific domain of speech generation, potentially missing out on speech-specific semantic nuances that could be learned jointly. The improvement in WER/CER, while statistically significant, is not massive, and the gains in perceptual quality (UTMOS) are marginal.
SemBridge contributes to the field of speech synthesis by providing a robust method for improving content fidelity in continuous-latent models. This can lead to more reliable and accurate text-to-speech and singing voice synthesis systems, which have broad applications in accessibility, entertainment, and human-computer interaction. The method's ability to improve lyric intelligibility in SVS is particularly relevant for the music industry. However, the potential for misuse in generating deepfakes or misleading audio content remains a concern, as with any advanced speech synthesis technology. The paper does not explicitly discuss safety mitigations, which is a common omission in technical research but important for broader impact assessment. SemBridge introduces a training-only semantic-token anchoring framework that improves content fidelity in continuous-latent autoregressive speech generation by supervising LM states with discrete semantic tokens, demonstrating significant gains in WER/CER across TTS and SVS benchmarks while maintaining competitive acoustic quality.
Mechanistic interpretability of music generation has concentrated on audio models, leaving symbolic models largely unexplored. We analyze two public text-to-MIDI systems of contrasting design: the purpose-built encoder--decoder text2midi and MIDI-LLM, a Llama~3.2~1B model extended with MIDI tokens using linear probing, the logit and tuned lenses, activation patching and difference-in-means steering. Across these methods, we recover musically meaningful structure and show how architecture shapes its formation and control. Pitch, instrumentation, harmony and texture are linearly decodable in both models. text2midi refines predictions gradually across depth, whereas MIDI-LLM works largely in its inherited textual basis before a sharp late rotation into the musical vocabulary; patching identifies a matching late attenuation of prompt-driven instrument transfer. Steering produces bidirectional changes in register and polyphony in both systems, and in tempo/energy in MIDI-LLM. Our two-orientation protocol isolates directional control and shows that all-layer interventions are robust in text2midi but accumulate disruptively in MIDI-LLM. Together, the results provide a practical toolkit for tracing and controlling musical concepts in symbolic generators. Audio examples are available on a demo website.
Primary: Warsaw University of Technology
All Institutions: Warsaw University of Technology
This paper makes a significant contribution to the field of mechanistic interpretability by extending its methods to symbolic music generation, offering a detailed comparative analysis of two distinct architectures and providing a robust protocol for causal intervention and control.
The paper employs a rigorous mechanistic interpretability toolkit, including linear probing, logit/tuned lenses, activation patching, and difference-in-means steering. The methodology is sound and well-executed, particularly the novel "bidirectional evaluation protocol" for steering, which effectively decomposes directional control from symmetric drift. The comparative analysis between a purpose-built encoder-decoder (text2midi) and a repurposed LLM (MIDI-LLM) provides valuable insights into how architecture influences the formation of symbolic representations. The use of the tuned lens to disentangle basis changes from genuine prediction formation in the LLM is a strong technical contribution.
The experimental setup is comprehensive, covering two distinct models and multiple interpretability techniques. The results are clearly presented and support the claims of architectural differences in prediction formation (gradual vs. late binding). The steering experiments demonstrate robust control over register and polyphony, with quantifiable metrics for tempo/energy in the LLM. The controlled probing on SynTheory adds necessary isolation to the ecological validity of full-generation probing. The statistical treatment, including confidence intervals and seed-clustered standard errors, is appropriate.
The paper provides detailed descriptions of the models, datasets (MIDICaps, SynTheory), and experimental protocols. The authors state that code will be released, and audio examples are available online. The specific hyperparameters for probing and steering are reported. However, the exact codebase for the "tuned lens" implementation and the specific steering vector computation details could benefit from more explicit pseudocode or references to specific libraries to ensure perfect reproducibility.
The study is limited to two models, which restricts the generalizability of the architectural conclusions. The metrics for tempo and polyphony are proxies (note density, mean pitch) and may not perfectly capture the musical concepts, especially in multi-track MIDI where interactions are complex. The probing labels for full generations are heuristic estimates, introducing potential noise. The steering directions are derived from text contrasts that vary lexically and musically, making it difficult to isolate the pure musical concept vector.
This work bridges the gap between mechanistic interpretability in language models and symbolic music generation. It provides a practical toolkit for researchers and developers to trace and control musical concepts in generative models, potentially leading to more controllable and interpretable music AI systems. The findings on architecture-dependent intervention strategies are directly applicable to the design of future symbolic music models. This paper makes a significant contribution to the field of mechanistic interpretability by extending its methods to symbolic music generation, offering a detailed comparative analysis of two distinct architectures and providing a robust protocol for causal intervention and control.
Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance. Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript. Both tasks require the generated speech to express the intended words while remaining consistent with the surrounding speaker identity, prosody, timing, and recording conditions. Discrete diffusion is particularly well suited to these tasks because it can iteratively refine masked tokens while jointly conditioning on both left and right acoustic context. We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical codec tokens. Its core architecture, HiCoDD, follows the RVQ generation order by representing previously generated codebooks as clean, committed acoustic context and applying diffusion only to the current refinement codebook. This separation enables leakage-free joint training while matching sequential coarse-to-fine inference. The model further combines phoneme-level conditioning, span-localized classifier-free guidance, and duration prediction to support both fixed-duration inpainting and variable-duration text edits. On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods. It also outperforms the evaluated autoregressive baselines across all speech-inpainting settings, on both single and multiple gaps. These results demonstrate that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. See our full code at https://github.com/iftachShoham/SIEDD.
Primary: Ben-Gurion University of the Negev
All Institutions: Ben-Gurion University of the Negev, University of Haifa
The paper presents SIEDD, a novel discrete diffusion framework that leverages the hierarchical structure of neural codecs for superior text-guided speech inpainting and editing. By introducing HiCoDD, which enforces coarse-to-fine generation and leakage-free conditioning, the authors address key limitations of existing autoregressive and naive diffusion approaches, achieving state-of-the-art performance on the RealEdit benchmark.
The paper proposes SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing. The core methodological contribution is HiCoDD, a hierarchical diffusion process that respects the Residual Vector Quantization (RVQ) structure of neural codecs (like EnCodec). Instead of flattening all codebooks into a single sequence, HiCoDD generates codebooks sequentially from coarse to fine, conditioning the diffusion on previously committed (clean) lower-level codebooks. This addresses the issue of "leakage" and ensures that the hierarchical dependency of the codec is preserved during generation. The paper also introduces span-localized classifier-free guidance (CFG) adapted for categorical scores, which randomizes phonemes only within the edited span to preserve surrounding context. A duration predictor is included to handle variable-length edits. The methodology is technically sound, well-motivated by the limitations of autoregressive models in non-causal generation tasks, and correctly applies discrete diffusion theory (DWDSE) to the hierarchical token space.
The evaluation is conducted on the RealEdit benchmark, comparing SIEDD against VoiceCraft, SSR-Speech, and a TTS baseline. Metrics include WER, Speaker Similarity (SIM), Mel Cepstral Distortion (MCD), F0/Energy distance, and UTMOS. SIEDD achieves state-of-the-art results in overall editing performance (lowest WER, highest SIM, lowest MCD) and significantly outperforms baselines in speech inpainting, particularly in multi-gap scenarios where autoregressive models struggle with error propagation. The ablation studies effectively isolate the contributions of the hierarchical structure, duration predictor, and localized CFG. The results are consistent with the claims, showing that modeling the codec hierarchy improves context preservation. The use of standard, widely accepted metrics strengthens the validity of the claims.
The paper provides a GitHub link to the code and configuration files. The appendix details the architecture, training hyperparameters, and dataset construction for the duration predictor. The training setup (550k steps on two A6000 GPUs) is specified. The use of standard datasets (LibriTTS, GigaSpeech) and tokenizers (EnCodec) aids reproducibility. The detailed ablation study configurations further support reproducibility.
The method relies on the quality of the underlying neural codec (EnCodec); errors in tokenization may propagate. The diffusion process, while effective, is computationally more expensive than autoregressive inference, although inference time is not the primary focus of this work. The duration predictor, while better than heuristics, may still struggle with extreme prosodic variations or non-speech content. The evaluation is limited to English speech.
This work contributes to the field of generative audio, specifically in robust speech editing and restoration. Applications include audio repair, content creation, and accessibility. The ability to edit speech with high fidelity and speaker consistency has significant implications for media production and communication. However, as with all speech editing technologies, there are potential risks regarding misinformation and deepfakes, necessitating responsible use and detection methods. The paper presents SIEDD, a novel discrete diffusion framework that leverages the hierarchical structure of neural codecs for superior text-guided speech inpainting and editing. By introducing HiCoDD, which enforces coarse-to-fine generation and leakage-free conditioning, the authors address key limitations of existing autoregressive and naive diffusion approaches, achieving state-of-the-art performance on the RealEdit benchmark.
Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.
Primary: Shanghai Innovation Institute
All Institutions: Shanghai Innovation Institute, Shanghai Jiao Tong University
GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
The paper proposes GROW, a reinforcement learning method tailored for autoregressive-diffusion Text-to-Speech (TTS) models, specifically instantiated on DiTAR. The core innovation lies in addressing the inefficiency and instability of converting deterministic ODE sampling (used in flow-matching) into Stochastic Differential Equations (SDEs) for policy gradient methods. Instead, GROW applies advantage-weighted regression directly to the flow-matching objective. It samples a group of on-policy rollouts, computes group-relative advantages for intelligibility and speaker similarity rewards, and uses these signed, normalized advantages to weight the standard flow-matching loss. A Wasserstein-2 velocity penalty is used to anchor the model to the pretrained reference, preventing divergence. The methodology is theoretically grounded, drawing on Taylor expansions to justify why linear advantage weighting outperforms exponential weighting in the high-reward, concentrated regime typical of strong pretrained TTS models. The approach is elegant in its simplicity, avoiding the overhead of trajectory storage and per-step likelihood ratio calculations required by methods like GRPO or PPO in continuous spaces.
The experimental evaluation is comprehensive and robust. The authors compare GROW against a strong baseline (DiTAR-GRPO) and the pretrained model across three benchmarks: LibriSpeech-PC, Seed-TTS EN, and Seed-TTS ZH. Results show consistent improvements in Word Error Rate (WER) and Speaker Similarity (SIM) while maintaining or slightly improving UTMOS. Notably, GROW achieves these gains with significantly higher training efficiency, being 2.9x faster than the 32-NFE DiTAR-GRPO baseline when using 10-NFE rollouts. The ablation studies are particularly strong, providing detailed analysis on the impact of rollout NFE, CFG settings, learning rates, W2 anchor strength, and reward design. The decomposition of self-imitation vs. reward-contrast provides valuable insight into why the proposed weighting scheme works better than exponential alternatives.
The paper provides extensive implementation details, including model architecture (Qwen3-0.6B backbone, LocDiT), hyperparameters (learning rates, group size, NFEs), and dataset descriptions. The authors explicitly state they will open-source the code, reproduction, and checkpoints, which significantly enhances reproducibility. The algorithm is clearly defined with pseudocode. The use of standard metrics (WER, SIM, UTMOS) and public datasets further supports reproducibility.
The method is currently evaluated primarily on zero-shot TTS tasks using specific reward models (ASR for WER, Speaker Encoder for SIM). The generalizability to other TTS objectives (e.g., emotion, prosody) or other flow-matching architectures (e.g., F5-TTS) is suggested but not extensively demonstrated. The reliance on frozen reward models introduces potential biases inherent in those models (e.g., ASR errors affecting WER rewards). Additionally, the performance gain, while consistent, is modest in absolute terms (e.g., WER reduction of ~0.45), though significant in the context of already high-performing models. The method's effectiveness on very short or very long utterances is not explicitly analyzed.
This work contributes to the efficient alignment of generative audio models, potentially lowering the computational cost of post-training for high-quality TTS systems. This could democratize access to advanced speech synthesis technologies. However, as with any TTS advancement, there are risks associated with improved voice cloning capabilities, including potential misuse for deepfakes or misinformation. The authors' open-source commitment promotes transparency and further research in this area. GROW introduces a novel, efficient, and theoretically motivated advantage-weighted regression framework for flow-matching TTS, demonstrating superior performance and training efficiency over trajectory-based RL methods. The paper makes a significant technical contribution by bridging the gap between discrete-token RL alignment techniques and continuous flow-matching generators, offering a scalable solution for post-training autoregressive-diffusion models.
What is music style? Though often described using text labels such as "swing," "classical," or "emotional," the real style remains implicit and hidden in concrete music examples. In this paper, we introduce a cross-modal framework that learns implicit music styles from raw audio and applies them to symbolic music generation. Inspired by BLIP-2, our model leverages a Querying Transformer (Q-Former) to extract style representations from a large, pre-trained audio language model (LM), and further applies them to condition a symbolic LM for generating piano arrangements. We adopt a two-stage training strategy: contrastive learning to align auditory style with symbolic expression, followed by generative modeling for music arrangement. Our model generates piano performances jointly conditioned on a lead sheet (content) and a reference audio example (style), enabling controllable and stylistically faithful arrangement. Experiments demonstrate the effectiveness of our approach in piano cover generation, style transfer, and audio-to-MIDI retrieval, achieving substantial improvements in style-aware alignment and music quality.
Primary: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
All Institutions: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), National University of Singapore, School of Computing
The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
The paper proposes a cross-modal framework for piano arrangement by leveraging a Querying Transformer (Q-Former) to bridge audio and symbolic domains. The core innovation lies in using the Q-Former to extract implicit style representations from a frozen audio language model (MusicGen) and conditioning a symbolic language model (MuseCoco) with these representations, alongside a lead sheet for content. The two-stage training processâcontrastive/matching learning for style alignment followed by generative modelingâis a logical extension of vision-language bootstrapping methods (like BLIP-2) to the audio-symbolic domain. The approach effectively disentangles content (lead sheet) from style (audio reference), addressing a significant gap in current music generation systems that often rely on text prompts or explicit symbolic controls. The use of LoRA adapters to condition the frozen symbolic LM is a practical and efficient design choice.
The experimental evaluation is comprehensive, covering objective metrics (MCA, CA, GPC, VCC, TA) and subjective listening tests. The inclusion of out-of-distribution datasets (Ballroom, GTZAN) strengthens the claim of generalization. The results show competitive performance against baselines like PiCoGen2 and A2M, particularly in style coherence metrics. The audio-to-MIDI retrieval task further validates the quality of the learned cross-modal representations. However, the lower content preservation scores on the in-distribution POP909 set compared to PCG2 are a notable weakness, attributed to errors in the upstream Sheetsage transcription. The subjective evaluation, while well-designed with ANOVA analysis, relies on a small sample size (21 participants), which limits the statistical power of the findings.
The paper provides detailed model configurations, including parameter counts, training epochs, batch sizes, and optimizer settings. The use of open-source or widely available backbones (MusicGen, MuseCoco, Sheetsage) enhances reproducibility. The data pairing strategy and specific metric definitions are clearly described. The lack of a public code repository is a minor drawback, but the detailed appendix and demo page mitigate this to some extent.
The authors acknowledge that the style representation is segment-level (4-bar), which may not capture long-term stylistic evolution or transitions. The reliance on Sheetsage for lead sheet extraction introduces potential error propagation. The method is currently limited to piano arrangement, and generalizing to multi-track or more complex instrumental arrangements is identified as future work. The subjective evaluation sample size is small.
This work contributes to the field of AI-assisted music creation by providing a tool for expressive piano arrangement. It enables musicians to reinterpret existing audio with specific stylistic nuances while maintaining harmonic and melodic integrity. The cross-modal representation learning technique has broader applications in audio-symbolic retrieval and style transfer. The potential for misuse in generating copyrighted music arrangements without permission is a consideration, though the focus on arrangement rather than direct cloning mitigates some risks. The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.
Primary: Queen Mary University of London
All Institutions: Queen Mary University of London, UKRI Centre for Doctoral Training (CDT) in Artificial Intelligence and Music (AIM)
The paper presents a solid technical contribution by introducing an equivariance regularization loss for music transformers, demonstrating that explicit inductive biases improve both equivariance and generative performance, and revealing that equivariance does not emerge naturally with scale.
The paper proposes the Equivariant Music Transformer (EMT), which introduces an auxiliary equivariance regularization loss to a standard next-token prediction transformer. The method involves a self-distillation setup where a shifted version of the input is processed, and the output distribution is aligned with the shifted original distribution using KL divergence. The authors argue that this enforces latent space equivariance. The approach is conceptually sound and addresses a specific gap in symbolic music generation: the lack of explicit inductive biases for pitch and time translational symmetry. However, the novelty is somewhat incremental, as equivariance has been explored in other domains (e.g., vision, NLP) and feature-level equivariance (FME/MRA) is already established in music transformers. The contribution lies primarily in the specific application of this regularization to generative music transformers and the empirical analysis of how equivariance degrades with scale.
The experimental evaluation is comprehensive. The authors conduct ablation studies comparing their method against data augmentation, feature-level equivariance, and a combination of both. They also benchmark against strong external baselines (Anticipatory Music Transformer, MIDI-LLM). The key findingâthat standard transformers become *less* equivariant as they scaleâis an interesting empirical observation that challenges the assumption that equivariance emerges naturally. The objective metrics (cross-entropy loss, equivariance loss, Top-1/Top-5 accuracy) show consistent improvements. The subjective listening test (MOS) adds valuable human-centric validation, demonstrating that EMT maintains quality under shifted prompts better than baselines. The use of a paired Wilcoxon signed-rank test for statistical significance is appropriate.
The paper provides detailed implementation details, including dataset preprocessing (LMD clean), model architecture (12 layers, 768 hidden size), training hyperparameters (AdamW, learning rate schedule, loss weights), and evaluation protocols. The code and demos are available online, which significantly aids reproducibility. The description of the self-distillation mechanism and the specific transformations (pitch/time shifts) is clear.
The paper acknowledges that the equivariance loss acts as a regularizer but does not deeply analyze the potential trade-offs or failure modes. For instance, if the shift distribution in the loss does not match the shift distribution in the test data, performance might degrade. The reliance on LakhMIDI, which has known quality issues (noise, transcription errors), is a limitation, although the authors filter the data. The claim that "standard language modeling methods alone do not capture music's translational symmetries" is supported, but the extent to which this applies to non-music domains or other symbolic domains is not discussed. The computational cost of the auxiliary branch (doubling forward passes) is noted but not extensively analyzed in terms of efficiency gains vs. performance.
This work contributes to the broader field of AI for Music (AIM) by providing a robust method for generating symbolic music that respects musical structure (transposition/tempo). This can improve the usability of AI music assistants, allowing users to transpose or shift prompts without losing generation quality. It also highlights the importance of inductive biases in deep learning for structured data. The ethical statement correctly addresses copyright concerns associated with training on LakhMIDI. The paper presents a solid technical contribution by introducing an equivariance regularization loss for music transformers, demonstrating that explicit inductive biases improve both equivariance and generative performance, and revealing that equivariance does not emerge naturally with scale.
Text-to-music language models begin with a choice usually made by default: how to tokenize music. Normally entangled with backbone, data, and recipe, its effect has never been measured in isolation. We fix pretrained Qwen3.5 (0.8B-27B), data, budget, and decoding, and swap only the representation across seven tokenizations, anchoring texture metrics to each representation's model-free ceiling. The ordering is clean and surprising: representation, not model size, is the binding variable for distributional fidelity. Scaling the backbone 34x barely moves Frechet Music Distance (FMD), whereas switching representation halves it. PMT, a performance-resolution stream we release (10 ms timing, per-note velocity, multi-track texture; 609 symbols), reaches FMD 159 at 0.8B against 272-286 for beat grids (1.7-1.8x lower, up to 2.8x elsewhere; non-overlapping bootstrap CIs), so a 0.8B performance-resolution model beats a 27B beat grid. It reappears on a 26M from-scratch backbone and a second performance-resolution tokenizer: a property of the class, not one lucky vocabulary. Nor is it a finer-lattice artifact: snapping PMT's onsets to the beat grids' resolution still leaves it 67-129 FMD ahead of both (n=500). The effect is distributional; whether it is audible is a separate question, left open by our probe, with a human study pre-registered. Native caption adherence is weak but separable: a lightweight decode-time constraint doubles instrument-F1 (.28 to .60) and Correct-Key (.16 to .35) at no distributional cost. We release the harness, 25+ checkpoints, two corpora (86.6k aligned across caption/MIDI/ABC/audio; 6.25M captioned, the largest for music), and an imprinting diagnostic: published text-to-MIDI systems reproduce their training distribution near-invariant to the caption (72% vs. 71% chord-time on disjoint domains). The field's next representation claim can now be measured, not asserted.
Primary: Unknown (Affiliations not explicitly named in text, only numbered)
All Institutions: Unknown
The paper provides a rigorous, controlled evaluation demonstrating that token representation design is more critical than model scale for distributional fidelity in text-to-symbolic-music generation, supported by the release of new datasets and benchmarks.
The paper proposes "PMT" (Performance-timed Music Tokens), a tokenization scheme for symbolic music generation that preserves 10ms micro-timing and per-note velocity, contrasting with traditional beat-grid tokenizations (like REMI) that quantize timing to metrical positions. The core methodological contribution is not just the tokenizer itself, but the rigorous "controlled swap" evaluation protocol. The authors fix the backbone (Qwen3.5, 0.8B-27B), data, training budget, and decoding strategy, varying only the tokenization scheme across seven different representations. This allows for an isolation of the representation's effect on distributional fidelity. The integration recipe involves vocabulary extension and caption-masked supervised fine-tuning. The approach is technically sound and addresses a significant confound in current music LLM research where representation, data, and model size are often entangled.
The experimental design is the strongest aspect of the paper. By controlling for model scale and data, the authors demonstrate that representation quality dominates model size for distributional fidelity (measured by FrĂŠchet Music Distance). A 0.8B model using PMT outperforms a 27B model using beat grids. They further validate this on a 26M parameter from-scratch backbone and a second performance-resolution tokenizer (PerTok), suggesting the result is a property of the representation class. They also provide an "imprinting diagnostic" showing that existing systems (MIDI-LLM, text2midi) reproduce training distributions rather than following captions, highlighting a limitation in current text-conditioned symbolic generation. The release of two large datasets (86.6k aligned, 6.25M captioned) and a benchmark harness adds substantial empirical value.
High. The authors release the PMT tokenizer, the evaluation harness, 25+ checkpoints, and the datasets. The controlled swap protocol is clearly defined, allowing other researchers to replicate the comparison. The use of standard backbones (Qwen3.5) and open metrics (FMD) further aids reproducibility.
The paper explicitly acknowledges that the distributional fidelity gains do not necessarily translate to perceptual/audible quality, which is left for a pre-registered human study. Additionally, caption adherence (e.g., key, instrument) is weak for PMT compared to dedicated systems, although the authors show a lightweight decode-time constraint can mitigate this. The "imprinting" finding suggests a broader issue with text-conditioned symbolic generation that may require architectural changes beyond tokenization to fully resolve. The lack of explicit institution names in the provided text makes institutional assessment difficult, though the technical content is strong.
This work has significant implications for the field of symbolic music generation and potentially other structured artifact generation domains (e.g., code, 3D assets). It challenges the assumption that scaling models alone will solve quality issues, emphasizing the critical role of representation design. The release of large, high-quality datasets and benchmarks will facilitate future research. The finding that representations can dominate scale is a crucial lesson for efficient model development. The paper provides a rigorous, controlled evaluation demonstrating that token representation design is more critical than model scale for distributional fidelity in text-to-symbolic-music generation, supported by the release of new datasets and benchmarks.
Reference-conditioned melody-preserving lyric editing replaces words while retaining a performance's timing, singer identity, and naturalness. Continuous-latent autoregression avoids finite codebooks and offers stepwise generation with learned stopping. Editing creates a conflict absent from ordinary reconstruction: training pairs reference cues with original lyrics, whereas inference asks revised lyrics to override source-lyric-correlated cues; one source-following patch can propagate through AR history. We introduce CLASVS. Its State-Control-Transition (SCT) routing keeps target-lyric and reference-melody controls persistent, returns semantic feedback on phonetic progress to the causal planner, and confines the previous latent patch to the local Transition. Progressive State-Control Grounding (PSCG) learns this contract through paired-edit-free, content-consistent Mandarin reconstruction. On two Mandarin benchmarks, CLASVS improves all four operations over discrete-AR Vevo2 and reduces macro-PER by 46.2%, while maintaining melody, singer similarity, and perceptual quality. Together, these results establish a strong continuous-AR operating point for score-annotation-free lyric edits and a basis for broader stepwise control. Audio demonstrations are available on our project page: https://piedpiperg.github.io/Liyric-SVS/.
Primary: Tencent AI Lab
All Institutions: Tencent AI Lab, Tsinghua University, Peking University
CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
The paper proposes CLASVS, a continuous-latent autoregressive model for melody-preserving lyric editing in singing voice synthesis. The core methodological contribution is the State-Control-Transition (SCT) routing mechanism, which attempts to resolve the conflict between preserving reference melody/timing and generating new lyrics. It separates the control signals (target lyrics, reference melody) from the generative state (semantic feedback, previous latent patch). The training strategy, Progressive State-Control Grounding (PSCG), relies on content-consistent reconstruction (training on the same audio/transcript) rather than paired counterfactual edits, which is a significant theoretical challenge. The use of a frozen causal semantic encoder for phonetic progress feedback is a novel architectural choice for maintaining lyric alignment in an autoregressive setting. However, the reliance on existing components (AudioVAE, Qwen3-0.6B, Flow-DiT) limits the novelty of the underlying generative primitives, focusing innovation primarily on the routing and training protocol.
The evaluation is comprehensive, comparing CLASVS against discrete-AR (Vevo2) and continuous-NAR (YingMusic-Singer-Plus) baselines. The paper reports significant improvements in macro-Phone Error Rate (PER) and perceptual metrics (MOS) over Vevo2. The inclusion of a "target-preference margin" metric to audit whether the model actually follows target lyrics rather than reverting to source lyrics is a strong methodological addition to the evaluation suite. The ablation studies effectively isolate the contributions of the SCT components. However, the evaluation is restricted to Mandarin, and the comparison with YingMusic+ shows mixed results (CLASVS is better at deletion/insertion, but YingMusic+ is better at substitution), suggesting that the autoregressive approach may have limitations in handling large lexical changes compared to parallel generation.
The paper provides detailed descriptions of the architecture, training stages, and hyperparameters. It mentions the release of code and checkpoints upon publication, which supports reproducibility. The use of standard datasets (Emilia, curated singing) and public baselines facilitates comparison. The specific "CLA-LyricEdit-320" benchmark is described in detail, allowing for independent verification.
The primary limitation is the lack of paired counterfactual training data, which forces the model to generalize from reconstruction tasks. This may limit its ability to handle complex edits or non-Mandarin languages. The autoregressive nature of the model results in slower inference speeds compared to NAR baselines like YingMusic+. The evaluation is limited to Mandarin, and cross-lingual generalization is not tested. Additionally, the reliance on automatic melody extraction (chroma-derived tokens) may introduce errors that propagate through the generation process.
This work contributes to the field of controllable singing voice synthesis, enabling more natural and flexible lyric editing without the need for manual score annotations. This has applications in music production, karaoke, and accessibility. However, the ability to edit lyrics while preserving singer identity raises ethical concerns regarding voice impersonation and copyright, which the authors acknowledge and address with proposed safety measures. CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.