Zero-shot Emotion Transfer for Cross-lingual Speech Synthesis

1. Abstract

Zero-shot emotion transfer in cross-lingual speech synthesis aims to transfer emotion from an arbitrary speech reference in the source language to the synthetic speech in the target language. Building such a system faces challenges of unnatural foreign accents and difficulty in modeling the shared emotional expressions of different languages. This paper based on DelightfulTTS addresses these challenges by leveraging different modules to model the language-specific prosody features and language-shared emotional expressions separately. Specifically, the language-specific speech prosody is learned by a non-autoregressive predictive coding (NPC) to improve the naturalness of the synthetic cross-lingual speech. The shared emotional expression between different languages is extracted from a pre-trained self-supervised model Hubert with strong generalization capabilities. We furthermore use hierarchical emotion modeling to capture more comprehensive emotions across different languages. Experimental results demonstrate the proposed framework's effectiveness in synthesizing bi-lingual emotional speech for the monolingual target speaker without emotional training data.

2. Demos

• M3: The structure of the model follows [1].
• SCLN-GST: The structure of the model follows [2].
• PROPOSED.

Synthesizing emotional speech for the target speaker in one language (e.g., Thai) without his/her emotional training data, where the emotion is transferred from a speech reference in another language (e.g., Chinese). Here Thai --> Chinese and Chinese -->Thai are both supported.