TTS Data Cleaning: Four Pipelines, Four Kinds of Dirty Work

Put four TTS data-cleaning pipelines side by side, and the differences are bigger than the names. In her tweet thread, Bella Chai breaks down Fish Audio S2, Emilia-Pipe, CosyVoice 3, and NeMo Curator. VAD cuts out human speech, diarization separates speakers, and ASR turns speech into text; these three are like the entry ticket. Where the audio comes from determines what additional checks are needed later.

Fish Audio S2 handles audio with background music and noise. Separating voice from background does not mean the audio is clean. After separation and VAD segmentation, it adds its own speech quality model, SQM, specifically to catch residual music, overlapping voices, and low-quality segments. Final transcription provides not only text but also speaker identity and tone labels. Training data needs more than matching words; the voice itself must be usable.

Emilia-Pipe deals with noisy audio scraped from the web. The pipeline first standardizes to 24 kHz mono, performs UV vocal separation, then uses pyanote for speaker diarization and Silero VAD for segmentation, followed by Whisper transcription. Next it checks DNSMOS, language confidence, and the ratio of average characters to phone duration. This ratio check is interesting: it can surface clips that sound acceptable in the recording but have suspicious transcripts. Web data is dirty, and many problems are not in the noise but in the mismatch between text and audio.

CosyVoice 3 focuses on multilingual data. Multilingual scenarios are prone to failing on transcription and punctuation. It uses multiple ASR models to cross-check transcripts, then uses MFA to align punctuation with actual pauses. Punctuation follows pauses, not text conventions. It also filters out major inconsistencies and finally checks volume and word-length ratio. Different models can produce different text for the same sentence, so cross-checking is unavoidable work.

NeMo Curator handles large mixed collections. It first normalizes and runs VAD, does a first round of quality checks, and makes bandwidth, UTMS, and SIGMOS thresholds configurable. Then it groups by speaker using concatenation and SortFormer, and re-checks VAD, bandwidth, and quality for each speaker. One person's voice may be clear while another person is far away and noisy; filtering the whole segment uniformly would cause collateral damage. It breaks checks down to the speaker level, maps original timestamps when exporting segments, and finally connects to ASR and text validation.

Looking at the four diagrams, there is no universal template for TTS data cleaning. Fish Audio S2 adds a quality model, Emilia-Pipe checks the text-audio ratio, CosyVoice 3 does multi-ASR cross-checking and pause-based punctuation, and NeMo Curator repeats quality checks per speaker. They are all answering the same question: in this batch of audio, which segments are truly usable for training? Different audio sources lead to different answers. Treat VAD, diarization, and ASR as the whole job, and you will most likely have to redo the work later.