Train multispeaker dataset wavernn

[JRMeyer]

>>> sanjaesc
[January 14, 2020, 5:30pm]

Hello everyone, I created a custom German dataset extracting audiofiles
from a game called Gothic. I've succesfully trained a model using the
repo from fatchord on one speaker.
Here you can hear samples from
the main hero of the game.

Any tips on how I would go ahead to train a multispeaker model? Would I
have to split every speaker into separate folders? Is there an entry
point I could start reading about this topic? slash
I'm also interested in training a vocoder, would it be possible to train
an universal vocoder explicit for this dataset?

Thanks for the repository and all the work you put into. slash
Best regards.



[Training universal PWGAN, background noise and bad
output

[This is an archived TTS discussion thread from discourse.mozilla.org/t/train-multispeaker-dataset-wavernn]

[JRMeyer]

>>> erogol
[January 15, 2020, 12:53am]

Sounds great. Is the game State of Mind? (Just curious)

TTS should already work for multi speaker. The best way is to format
your dataset as LibriTTS and use the formatter already there.

Regarding the vocoder. We already trained and released an universal
vocoder in WaveRNN repo https://github.com/erogol/WaveRNN . It works
for different speakers and languages nicely.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[January 27, 2020, 1:25pm]

The game series is called
Gothic.

I'm currently in my exam period, so I didn't have much time
experimenting.

I've trained a model to 130k steps, using Tacotron multi speaker and
GST. It learned a decent attention at 130k steps. I'll include more
speakers and fully train the model, once I have more time. Passing a
style_wav during inference, also changed the prosody of the generated
speech. I'm still not sure how to work with the Tokens from GST but I'll
read more about it, once the exams are over.

I haven't tried the universal vocoder yet, but my dataset has a SR of
22050. How would I go ahead and finetune the existing model?

I'll post the results later on.

Thanks and best regards.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[February 13, 2020, 11:29am]

Update.

12-07-47.filename}[1235×880 73.2

12-07-12.filename}[1467×896 308

I'm training on 10 speakers, using GST.

You can listen to some test sentences here.
Link slash
Without a style-wav the speech is way to fast.

I mostly used the default config.json from the master branch, also in
the link above. Any tips on the config? Should i tune some parameters?

At slash ~200k I accidently stopped the training and had to resume it, not
sure if it influenced it in a bad way.

I'll let it run for a little bit longer.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[February 14, 2020, 12:43pm]

Another Update:

Here
are some more examples from the multi speaker model. Generated at around
slash 320k steps. Also had the warning ' slash > Decoder stopped with
'max_decoder_steps'', but it fixed itself after some iterations.

...

On another note. If I want to extract the spectograms from the TTS
model, how would i go ahead doing so? The notebook
ExtractTTSpectrogram.ipynb
hast the following lines.

curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash # TODO: multiple speaker** slash
model = setup_model(num_chars, num_speakers=0, c=C)

Any tips on what changes are needed to extract mels from a multi speaker
model? slash
Help on this matter would be highly appreciated.

Best regards.

[Archived Post]

[JRMeyer]

>>> erogol
[February 14, 2020, 2:33pm]

Thx for sharing the results. It looks quite interesting and given the
figures, your training looks quite healthy.

For using the notebook, you need to give the right number of speakers to
the model and you need to provide the list of speakers. Speakers are
already exported to a json file. You need to load them and pass the list
to the model as well. If you look at the source code how it does the
inference, it is quite easy to implement it.

[Archived Post]

[JRMeyer]

>>> erogol
[February 14, 2020, 3:07pm]

You might also consider to set audio parameters. Because in general male
and female voices require different values. You can try
CheckSpectrograms notebook to optimize these values.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[February 16, 2020, 11:13am]

Thanks for the help! Modifying the notebook worked quite well once i
knew what to do. I'm currently retraining the model because it broke
after getting to r=1. It didn't recover from slash > Decoder stopped with
'max_decoder_steps' and broken even on short sentences. I also cleaned
up the dataset, since it had some samples where the speaker would laugh

Regarding the audio parameters, i used the notebook to evaluate some of
the values, but i would say the default values are good. Then again im
not 100% sure what to pay attention to. slash
The average of 'mel = AP.melspectrogram(wav)' fmel_min slash ~ 0.0000 and
fmel_max slash ~ 0.7900

[Archived Post]

[JRMeyer]

>>> sanjaesc
[February 19, 2020, 3:31pm]

Another Update: slash

16-05-34.filename}[1398×891 74.4

16-06-18.filename}[1463×918 299

As said im re-training the model. What i did now is:

And with forward_attention enabled during inference the attention
doesn't break even on long sentences.

Now i though about switching to BN prenet at around slash ~400k steps, based
on your comment
here. Does
it make sense?

I think once I'm done with this run, I'll go back and try r=1 one more
time.

PS: Some more examples
here.
Quite happy with the results.

[Archived Post]

[JRMeyer]

>>> nmstoker
[February 19, 2020, 8:35pm]

Did you get a sense that the memory_size changing to 5 made a
significant difference?

From the look of the lines on the charts they look fairly similar before
and after, but maybe they helped with qualities not apparent in the
charts alone.

I'm in a similar position with my training and am on the brink of
attempting the 400k switch to finish off the model on BN. It's still
running right now but based on the charts it actually seems to be doing
okay on r=1 (touch wood!), which had never happened with previous runs
with earlier versions of the code. Thanks for posting your progress -
I'll be doing the same once I've tried the BN switch.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[February 20, 2020, 9:20am]

I would say the only real difference I noticed was, that it stopped
producing slash > Decoder stopped with 'max_decoder_steps'. But if it's in
correlation with the memory_size change or me cleaning up the dataset, I
cannot tell.

I now made the switch to BN at slash ~410k steps, let's see how it performs.

I'll try the same for r=1 after the current run and post the results for
comparison.

[Archived Post]

[JRMeyer]

>>> erogol
[February 20, 2020, 4:18pm]

You should go with BN after you are sure with the attention performance
of the model.

But afais, attention looks working fine.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[March 10, 2020, 10:55pm]

Sooo long time no hear...

As previously said, i was about to train the model using BN. It didnt
work. slash
Tried the switch to r=1. Didn't work.

Cleaned up the dataset and retrained the model. Best so far is still at
r=2.

Trained the new MelGAN vocoder to 600k steps, using the mels from
multi-speaker model.

Here are some samples slash
https://soundcloud.com/sanjaesc-395770686/sets/multi-speaker-mozilla-tts-melgan-vocoder slash
and some more slash
https://soundcloud.com/sanjaesc-395770686/sets/multi-speaker-mozilla-tts-melgan-vocoder-2 slash
PS: All use the same style_wav during inference

++++++++++++++++

On another note... I also tried training WaveRNN but will get

File '/home/alexander/anaconda3/lib/python3.7/site-packages/torch-1.4.0-py3.7-linux-x86_64.egg/torch/utils/data/_utils/fetch.py', line 47, in fetch
return self.collate_fn(data)
File '/media/alexander/LinuxFS/Documents/PycharmProjects/TTS/WaveRNN/dataset.py', line 74, in collate
coarse = np.stack(coarse).astype(np.float32)
File '<array_function internals>', line 6, in stack
File '/home/alexander/anaconda3/lib/python3.7/site-packages/numpy/core/shape_base.py', line 425, in stack
raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

This user at github had the same issue

github.com/mozilla/TTS

#### Tacotron2 + WaveRNN experiments

date='2018-05-06' time='02:57:06' timezone='UTC'}

date='2019-06-05' time='09:49:40' timezone='UTC'}

Tacotron2: https://arxiv.org/pdf/1712.05884.pdf WaveRNN:
https://github.com/erogol/WaveRNN forked from
https://github.com/fatchord/WaveRNN The idea is to add Tacotron2 as
another alternative if it is really useful then the slash ...

[enhancementstyle='display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;'}
[experimentstyle='display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;'}

any tips?

PPS: Like here Contributing my german voice for
tts
i also noticed that the model has some problems with umlauts. It will
pronounce o as ö and vice versa. Are graphemes the better choice here?

Best regards.

[Archived Post]

[JRMeyer]

>>> petertsengruihon
[March 12, 2020, 9:46am]

Have you solved the
ValueError: all input arrays must have the same shape problem?

[Archived Post]

[JRMeyer]

>>> sanjaesc
[March 13, 2020, 9:14am]

Not rearlly... but training on bits seem to work.

[Archived Post]

[JRMeyer]

>>> petertsengruihon
[March 14, 2020, 7:46am]

thanks sanjaesc. I have
tried to train on 10 bits yesterday too. Hope that would work. Would you
mind sharing your log ?

[Archived Post]

[JRMeyer]

>>> sanjaesc
[March 14, 2020, 5:32pm]

I'm currently experimenting with pwgan and melgan. Afterwards I'll
retrain the tts model using graphemes and give wavernn a try.

[Archived Post]

[JRMeyer]

>>> petertsengruihon
[March 16, 2020, 2:42am]

. I have also tried
pwgan too based on erogol
project. May i ask which melgan repo that you tried ? I have tried this
melgan repo but hop size could
not change into 256, which is correspond to mozilla TTS.

[Archived Post]

[JRMeyer]

>>> sanjaesc
[March 16, 2020, 7:34am]

MelGAN is also included in the
pwgan
repo slash ^ slash ^. slash
Just use one of the melgan.yaml configs.

[Archived Post]

[JRMeyer]

>>> petertsengruihon
[March 16, 2020, 7:51am]

Oh i see. Really appreciate it. Thanks a lot.

[next page
→

[Archived Post]