RSE-TTS: Compact Text-to-Speech through Shared Text-Aware Reference Encoding and Generation
Abstract
Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its transcript further strengthens this capability. Controlled ablations and activation interventions trace the text benefit through intermediate acoustic representations into the compressed vectors that condition generation. Building on these findings, we introduce RSE-TTS, whose Reconstructive Speaker Encoder (RSE) shares the generation backbone and compresses paired reference text and speech into eight continuous vectors. Sharing reduces encoder-generator parameters by 48% relative to independent pretrained encoding. Large-scale English training yields a 118M-parameter generator that improves both word error rate and speaker similarity over the corresponding official MOSS-TTS-Nano cloning modes on all three English benchmarks. Combined with acoustic continuation, RSE-TTS approaches the speaker similarity of substantially larger systems while achieving lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local models. Audio samples are available at https://voxnexusai.github.io/RSE-TTS/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that a pretrained TTS backbone can be effectively repurposed as a text-aware reference encoder, significantly improving speaker similarity in compact models. By sharing the generation backbone and jointly encoding reference text and speech, RSE-TTS achieves a 48% reduction in parameters while outperforming baseline compact systems and approaching the quality of much larger models, supported by rigorous mechanistic analyses of the encoding process.
Methodology
The paper proposes RSE-TTS, a compact text-to-speech system that repurposes a pretrained autoregressive TTS backbone as a reference encoder. The core methodological contribution is the "Reconstructive Speaker Encoder" (RSE), which shares weights with the generation backbone to compress paired reference text and speech into eight continuous vectors. This parameter-sharing strategy reduces the encoder-generator parameter count by 48% compared to independent encoders. The authors provide a rigorous mechanistic analysis using activation patching and gradient interventions to demonstrate that the reference text influences intermediate acoustic representations, which are then captured by the query vectors. The approach effectively bridges the gap between generative modeling and speaker encoding, leveraging the text-speech alignment learned during pretraining to improve voice cloning fidelity in compact models.
Results & evaluation
The experimental evaluation is comprehensive, utilizing three major English benchmarks (Seed-TTS, CV3-eval, MiniMax). The authors conduct controlled ablations to isolate the effects of pretrained initialization, reference text inclusion, and parameter sharing. Results show that RSE-TTS outperforms the official MOSS-TTS-Nano in both Word Error Rate (WER) and Speaker Similarity (SIM). Furthermore, when combined with acoustic continuation, the system approaches the performance of much larger models like CosyVoice 3 and MOSS-TTS-Local while maintaining a significantly smaller footprint (129M vs. 859M+ parameters). The inclusion of mechanistic studies (e.g., reversing transcript order, patching activations) adds depth to the empirical validation, confirming the causal role of text-aware encoding in voice reconstruction.
Reproducibility
The paper provides detailed descriptions of the model architecture, training objectives, and evaluation protocols. However, the primary training data is an "in-house English speech corpus," which limits full reproducibility. The authors do not release the code or the specific checkpoint used for the final results, though they reference the base MOSS-TTS-Nano model. The detailed hyperparameters and loss weighting schemes in the appendix aid in partial reproduction, but the lack of open-source code and proprietary data remains a barrier.
Limitations
The system requires a reference transcript for optimal performance, which may not be available in all real-world scenarios. The paper focuses exclusively on English, leaving generalizability to other languages unverified. Additionally, the reliance on an in-house dataset for the final large-scale training limits the ability for the community to verify the scaling claims independently. The "acoustic continuation" mode, while effective, increases inference latency and context window requirements, partially negating the efficiency gains of the compact encoder.
Broader impact
This work contributes to the development of efficient, high-fidelity voice cloning systems suitable for edge devices or real-time applications. By demonstrating that generative capabilities can be transferred to encoding tasks, it offers a new paradigm for designing compact multimodal systems. The mechanistic insights into how text influences speaker representations could inform future architectures for zero-shot TTS and voice conversion. The paper demonstrates that a pretrained TTS backbone can be effectively repurposed as a text-aware reference encoder, significantly improving speaker similarity in compact models. By sharing the generation backbone and jointly encoding reference text and speech, RSE-TTS achieves a 48% reduction in parameters while outperforming baseline compact systems and approaching the quality of much larger models, supported by rigorous mechanistic analyses of the encoding process.
Author of this paper? Send a correction.