Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Hackathon 7th] fix Tacotron2 with CSMSC #3905

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

[Hackathon 7th] fix Tacotron2 with CSMSC #3905

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6dd39a2
fix Tacotron2 with CSMSC
Liyulingyue Nov 22, 2024
85e8c4e
Update preprocess.py
Liyulingyue Nov 25, 2024
fe64b19
Update README.md
Liyulingyue Nov 30, 2024
e997640
Merge branch 'develop' into csmsc-tts0
Liyulingyue Dec 2, 2024
e137dbd
Merge branch 'develop' into csmsc-tts0
Liyulingyue Dec 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions examples/csmsc/tts0/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ This example contains code used to train a [Tacotron2](https://arxiv.org/abs/171
### Download and Extract
Download CSMSC from it's [Official Website](https://test.data-baker.com/data/index/TNtts/) and extract it to `~/datasets`. Then the dataset is in the directory `~/datasets/BZNSYP`.

The structure of the folder is listed below.

```text
datasets/BZNSYP
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datasets/BZNSYP should have three folders:

└─ Wave
    └─ .wav files (audio speech)
└─ PhoneLabeling
    └─ .interval files (alignment between phoneme and duration)
└─ ProsodyLabeling
   └─ 000001-010000.txt (text with prosodic by pinyin)

Still we only use .wav files in training.

└── Wave
└── .wav files
```

### Get MFA Result and Extract
We use [MFA](https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner) to get phonemes for Tacotron2, the durations of MFA are not needed here.
You can download from here [baker_alignment_tone.tar.gz](https://paddlespeech.bj.bcebos.com/MFA/BZNSYP/with_tone/baker_alignment_tone.tar.gz), or train your MFA model reference to [mfa example](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/other/mfa) of our repo.
Expand Down
3 changes: 3 additions & 0 deletions paddlespeech/t2s/modules/nets_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,9 @@ def make_pad_mask(lengths, xs=None, length_dim=-1):
if length_dim == 0:
raise ValueError("length_dim cannot be 0: {}".format(length_dim))

# check if ilens is 0-dim tensor, if so, add a dimension
if lengths.ndim == 0:
lengths = lengths.unsqueeze(0)
bs = paddle.shape(lengths)
if xs is None:
maxlen = paddle.cast(lengths.max(), dtype=bs.dtype)
Expand Down
3 changes: 2 additions & 1 deletion paddlespeech/t2s/modules/tacotron2/attentions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,8 @@ def forward(
if paddle.sum(att_prev) == 0:
# if no bias, 0 0-pad goes 0
att_prev = 1.0 - make_pad_mask(enc_hs_len)
att_prev = att_prev / enc_hs_len.unsqueeze(-1)
att_prev = att_prev / enc_hs_len.unsqueeze(-1).astype(
att_prev.dtype)

# att_prev: (utt, frame) -> (utt, 1, 1, frame)
# -> (utt, att_conv_chans, 1, frame)
Expand Down
3 changes: 3 additions & 0 deletions paddlespeech/t2s/modules/tacotron2/encoder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,9 @@ def forward(self, xs, ilens=None):
return xs.transpose([0, 2, 1])
if not isinstance(ilens, paddle.Tensor):
ilens = paddle.to_tensor(ilens)
# check if ilens is 0-dim tensor, if so, add a dimension
if ilens.ndim == 0:
ilens = ilens.unsqueeze(0)
xs = xs.transpose([0, 2, 1])
# for dygraph to static graph
# self.blstm.flatten_parameters()
Expand Down