Audio ML PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-17

Back to Mel: Learning to Bridge Fidelity and Perception in Generative Speech Enhancement

Andong Li, Kun Zhou, Xiaodong Li, Chengshi Zheng

Institute of Acoustics, Chinese Academy of Sciences +5 · arXiv

Affiliations: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Télécom Paris, Institut Polytechnique de Paris

Abstract

Recent generative techniques have facilitated the rapid proliferation of speech enhancement approaches built upon learnable compact latent representations and large generative backbones. Nevertheless, it remains inherently challenging to balance between acoustic fidelity, perceptual quality, and processing efficiency. In this paper, we revisit the Mel spectrum as a structured acoustic representation and propose BAME, a flow-matching-based framework that exploits the complementary priors for generative speech enhancement task. First, leveraging the explicit hierarchical frequency distribution of Mel spectrum, we elaborately devise a subband-aware Transformer (SaT) for efficient generative modeling. Using only 2.9% of the parameters, the proposed SaT achieves superior enhancement performance compared with a 331 M DiT backbone. Second, motivated by range-null decomposition theory of Mel filtering in the magnitude-spectral domain, we introduce a phase-aware neural vocoder that incorporates degraded-phase prior, together with a quality-guided dual-phase strategy to adaptively mitigate the effects of unreliable phase information. Third, we devise a simple yet effective one-step fidelity-perception alignment strategy that fine-tunes the flow model with complementary constraints from fidelity and perception domains. Extensive experiments are conducted on DNS-Challenge, EARS, GSR, and URGENT benchmarks, and both objective and subjective results demonstrate promising performance on various intrusive and non-intrusive evaluation metrics. These results further advocate the potential of exploiting structured acoustic representations for reliable and efficient generative speech enhancement. Audio samples and codes are available at https://github.com/AndongLi-speech/BAME.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper proposes BAME, a flow-matching-based generative speech enhancement framework that leverages the structural priors of the Mel spectrum through a subband-aware Transformer and a phase-aware vocoder, achieving efficient one-step generation with balanced fidelity and perceptual quality. The technical contribution is significant due to the efficient backbone design, the rigorous exploitation of Mel's mathematical properties (RND), and the effective one-step fine-tuning strategy that addresses the fidelity-perception trade-off.

Methodology

The paper proposes BAME, a flow-matching-based framework for generative speech enhancement that revisits the Mel spectrum as a structured acoustic representation. The core novelty lies in the Subband-aware Transformer (SaT), which exploits the hierarchical frequency distribution of Mel bands to achieve efficient generative modeling with significantly fewer parameters (2.9% of a DiT baseline) while maintaining superior performance. The methodology also introduces a phase-aware neural vocoder that leverages the Range-Null Decomposition (RND) theory to recover magnitude information from Mel spectra and incorporates degraded phase priors. A key contribution is the Quality-guided Dual-Phase (QDV) strategy, which adaptively selects between zero-phase and degraded-phase vocoding based on non-intrusive quality scores to mitigate the instability of degraded phase in reverberant conditions. Finally, the One-Step Fidelity-Perception Alignment (OS-FPA) strategy fine-tunes the flow model to balance intrusive fidelity and non-intrusive perceptual quality in a single step, addressing the trade-off typically observed in iterative sampling methods.

Results & evaluation

The experiments are extensive, covering four major benchmarks: DNS-Challenge 2020, EARS, GSR, and URGENT. The evaluation includes a comprehensive suite of intrusive metrics (PESQ, eSTOI, LPS, SpkSim, SBS, WER) and non-intrusive metrics (DNSMOS, UTMOS, DiMOS, WVMOS, SCOREQ). Subjective A/B preference tests were conducted with 22 participants. Ablation studies thoroughly investigate the impact of the subband structure, the choice of backbone modules (GRU vs LSTM vs Conv), the trajectory type (FM vs Schrödinger Bridge), and the specific components of the OS-FPA strategy. The results demonstrate that the proposed SaT backbone outperforms larger DiT backbones and that the QDV strategy effectively improves robustness against phase artifacts.

Reproducibility

The paper provides a GitHub link with code and audio samples. Detailed hyperparameters, dataset configurations, and training procedures are described in the appendix and main text. The use of standard components (GRU, MHSA, WavLM) and open-source datasets enhances reproducibility. The specific implementation of the RND-based vocoder and the OS-FPA fine-tuning procedure are clearly defined.

Limitations

The method relies on a pre-trained WavLM model for semantic conditioning, which adds computational overhead during inference if not optimized. The QDV strategy requires running the vocoder twice and evaluating quality scores, which may increase latency compared to single-pass vocoding. The performance gains in some non-intrusive metrics are marginal compared to state-of-the-art generative methods, suggesting diminishing returns in perceptual quality. The reliance on Mel spectra may still limit the recovery of very fine-grained high-frequency details compared to complex spectrum modeling, although the phase-aware vocoder mitigates this.

Broader impact

This work advocates for the use of structured, handcrafted representations (Mel) over learned latents in generative speech enhancement, offering a more efficient and interpretable alternative. The subband-aware modeling approach could be extended to other audio processing tasks. The one-step generation strategy is particularly relevant for real-time applications where latency is critical. The integration of fidelity and perception alignment provides a framework for balancing objective reconstruction error with subjective quality, which is applicable to other generative audio tasks. The paper proposes BAME, a flow-matching-based generative speech enhancement framework that leverages the structural priors of the Mel spectrum through a subband-aware Transformer and a phase-aware vocoder, achieving efficient one-step generation with balanced fidelity and perceptual quality. The technical contribution is significant due to the efficient backbone design, the rigorous exploitation of Mel's mathematical properties (RND), and the effective one-step fine-tuning strategy that addresses the fidelity-perception trade-off.

Author of this paper? Send a correction.