Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions
Abstract
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces DiCoP and Pandemonium, which integrate diarization conditioning into a long-form Parakeet backbone with cross-speaker attention to improve target-speaker ASR in multi-talker scenarios. This work addresses significant structural limitations of previous diarization-conditioned ASR systems by enabling joint optimization and cross-speaker interaction modeling, supported by a robust pseudo-labeling pipeline for data scaling, resulting in state-of-the-art performance on meeting benchmarks.
Author of this paper? Send a correction.