Holistic Parallel Supervision for Expressive Speech-to-Speech Translation

VC-Dub: Dubbing + Voice Conversion

Professional dubbing supplies human-performed correspondence in content, prosody, and timing. Multilingual voice conversion (Seed-VC) moves the English dub to the original speaker's voice, so the resulting pairs are parallel in content, prosody, timing, and speaker identity. The original German/Spanish recording is retained as the target after audio cleaning, without TTS or VC resynthesis.

VC-DUB Cleaning and Filtering

We apply denoising, vocal extraction, language identification, speaker filtering, and speech-quality filtering before constructing the final VC-DUB training splits.

Note. Total hours denote the combined source and target durations, computed using the same VAD-span definition as Table 1. Retention is computed relative to the original raw dubbing pool. The final training-split row is included for comparison with Table 1 and does not represent an additional cleaning stage. The DNSMOSPro boundaries are observed after scale-matched selection, not preset thresholds.

Pair-Level Correspondence

Raw-Dub already gives stronger content, prosody, and timing correspondence than CVSS-T; its main weakness is speaker mismatch. Voice conversion substantially raises Vsim and further increases A.PCP. Shuffling the VC-Dub pairings (negative control) substantially reduces the correspondence scores.

Table 2. Pair-level correspondence and speech quality of the training data for CVSS-T, Raw-Dub, and VC-Dub. Shuffled VC-Dub pairs provide a negative control. DNSMOSPro is reported as source (En) | target (De/Es).

DirectS2ST

DirectS2ST is trained to predict all 16 target Mimi codec streams from source-derived conditions, with every stream supervised by the paired human recording. Its acoustic and speaker prompts are extracted from the source during both training and inference.

Results

504 test utterances per direction, evenly distributed across seven expressive conditions. Content: ASR-BLEU (Whisper-large-v3) and BLASER 2.0-QE; prosody and timing: AutoPCP (A.PCP), speaking-rate correlation (Rate) and pause similarity (Pause); speaker similarity: Vsim; naturalness: NISQA-TTS (NAT).

Table 3. Expressive S2ST results on mExpresso. ^† denotes significance between training sets within the same model, and ^‡ against the strongest baseline in the same language direction (two-sided paired randomization tests, 10,000 permutations, Holm–Bonferroni correction, p_adj < 0.05). Bold marks the best score per direction. Dub-S2ST is evaluated only on En→Es: the mHuBERT checkpoint used in this baseline covers English, Spanish, and French. CVSS-T^S is its speed-adapted CVSS-T variant. Dub-S2ST denotes our adapted baseline: we replace its original S2U module with the model of Popuri et al. (2022) and retain its CosyVoice-initialized U2S module. Both components are trained on the corresponding supervision dataset.

Table 4. Contribution of voice conversion: DirectS2ST trained on the same aligned dubbing pairs with (VC-Dub) and without (Raw-Dub) VC. VC mainly improves speaker similarity, with additional gains in A.PCP and Rate.

Speech Samples

Each system is trained on CVSS-T or on our VC-Dub data. Source utterances come from the mExpresso test set; texts are shown as in the dataset transcripts, with emphasized words in bold italics.

English → Spanish

The 10 utterances used in the expressiveness listening test (Table 5), with all six system configurations.

English → German

10 utterances from the En→De test set across expressive conditions. Dub-S2ST is not included: the mHuBERT checkpoint used in this baseline covers English, Spanish, and French.