Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer

Zhisheng Zhang, Yifan Mi, Yixuan Zhou, Haiyun Li, Shengbo Cai, Dongfu Song, Ziang Xu, Jie Hao, Zhiyong Wu

Shenzhen International Graduate School, Tsinghua University +3 · NeurIPS 2026

Affiliations: Shenzhen International Graduate School, Tsinghua University, Beijing University of Posts and Telecommunications

Paper Code Demo

Abstract

Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the semantic level: small perturbations can disrupt their semantic encoding process and consequently cause severe degradation in downstream ASR and LALM performance. To study this vulnerability, we propose T-SemAttack, a transferable semantic attack based on a semantic encoder ensemble. By jointly drifting the representation spaces of multiple semantic encoders, T-SemAttack effectively disrupts the semantic content of speech while preserving perceptual quality, thereby inducing error features. We further analyze layer-wise representation drift and cross-model gradient alignment. Our findings show that perturbations can induce substantial shifts in deep semantic representations and that the input gradients of different surrogate encoders exhibit directional alignment. These observations suggest that shared adversarially sensitive directions in the input space may be a key factor underlying cross-model attack transferability. Building on these, we introduce ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation to improve reconstruction fidelity and downstream SpeechLLM robustness. Extensive experiments on multiple datasets, 9 open-source and black-box ASR systems, and 4 LALMs demonstrate that T-SemAttack achieves strong transferable attack performance, while the proposed Robust Speech Tokenizer exhibits robustness under high-fidelity reconstruction and downstream tasks. Our code and demo are available at https://wxzyd123.github.io/t-semattack.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper identifies the speech tokenizer as a critical security bottleneck and proposes a transferable semantic attack (T-SemAttack) along with a robust tokenizer (ROSETok) to mitigate these vulnerabilities. The technical contribution is significant, combining a novel attack mechanism based on multi-encoder representation drift with a principled defense strategy involving semantic distillation, supported by extensive experiments across multiple ASR and LALM systems and rigorous human perceptual evaluations.

Methodology

The paper proposes T-SemAttack, a transferable semantic attack targeting the front-end speech tokenizer rather than the entire ASR/LALM pipeline. The core innovation is the use of a semantic encoder ensemble (S3Tokenizer, HuBERT, Whisper, MFCC) to jointly optimize perturbations that drift the representation space while preserving perceptual quality via a psychoacoustic masking constraint. This approach is methodologically sound, leveraging the shared structural bottlenecks (e.g., Mel-spectrogram inputs) of different encoders to achieve cross-model transferability. The proposed defense, ROSETok, employs a two-stage training strategy: base pretraining with noisy inputs and semantic distillation from a Whisper teacher, followed by attack-aware fine-tuning. The choice to target encoder-level semantic preservation rather than token-level invariance is well-justified by the analysis of how semantic drift, rather than mere token flipping, causes downstream failure.

Results & evaluation

The experimental evaluation is extensive and rigorous. The authors test the attack on 9 ASR systems (including black-box commercial APIs like GPT-4o-mini and Gemini) and 4 LALMs, demonstrating strong transferability. The inclusion of a human study (MOS evaluation) to verify perceptual quality is a significant strength, confirming that the attacks remain "harmless" to human listeners while being devastating to models. The evaluation of ROSETok shows clear improvements in reconstruction fidelity (STOI) and downstream semantic robustness (FSC scores) compared to baselines like DAC and StableToken. The ablation studies effectively isolate the contributions of the semantic loss, psychoacoustic constraint, and distillation components.

Reproducibility

The paper provides high reproducibility. Code and a demo are available. Detailed hyperparameters for both the attack (SGD steps, learning rate, perturbation budget) and the tokenizer training (batch size, learning rate schedules, loss weights) are specified. The datasets used (LibriTTS, Seed-TTS, Emilia) are standard or publicly available. The specific configuration of the surrogate encoders and the psychoacoustic model implementation details are described sufficiently for replication.

Limitations

The primary limitation is the computational cost of the attack, with an RTF of ~21.8, which is high for real-time applications, though the authors argue for an offline threat model. The defense (ROSETok) is evaluated primarily on Gaussian noise and the specific T-SemAttack; its robustness against other types of adversarial attacks or distribution shifts is not fully explored. The human study, while valuable, uses a small sample size (20 participants) and a limited set of utterances (4 speakers), which may not generalize to all demographic groups or acoustic conditions.

Broader impact

This work has significant implications for the safety and security of deployed speech AI systems. By identifying the tokenizer as a critical vulnerability point, it shifts the focus of security research from end-to-end model hardening to front-end robustness. The dual-use nature of the attack (potential for evading transcription or moderation) is acknowledged, but the defensive contribution (ROSETok) provides a constructive path forward. The findings are directly applicable to industry systems relying on speech tokenizers for LLM integration, urging a re-evaluation of their security posture. The paper identifies the speech tokenizer as a critical security bottleneck and proposes a transferable semantic attack (T-SemAttack) along with a robust tokenizer (ROSETok) to mitigate these vulnerabilities. The technical contribution is significant, combining a novel attack mechanism based on multi-encoder representation drift with a principled defense strategy involving semantic distillation, supported by extensive experiments across multiple ASR and LALM systems and rigorous human perceptual evaluations.

Author of this paper? Send a correction.