[Basic 5000] vits japanese model #1604

neurlang · 2022-05-27T03:34:05Z

neurlang
May 27, 2022

マッチ試験に備えて、本当に気合を入れて勉強しなきゃ。
Matchi shiken ni sonaete, hontōni kiai o irete benkyō shinakya.
mɑkki ʃikɛn ni sɔnɑɛtɛ, hɔntɔːni kiɑi ɔ irɛtɛ bɛnkjɔː ʃinɑkjɑ.

tts-output-4.mp4

カットの新決議案はかなりの衝撃を与える可能性もあります。
Katto no shin ketsugi-an wa kanari no shōgeki o ataeru kanōsei mo arimasu.
gɑttɔ nɔ ʃinkɛtsugi ɑn vɑ, kɑnɑri nɔ ʃɔːgɛki ɔ, ɑtɑɛru kɑnɔːsɛː gɑ ɑrimɑsu

tts-output-5.mp4

erogol · 2022-05-31T08:42:31Z

erogol
May 31, 2022
Maintainer

Sounds good! Is it the end result or still training?

0 replies

neurlang · 2022-06-01T02:19:08Z

neurlang
Jun 1, 2022
Author

this was step 60000, about 6 hour wallclock time.
it's sort of end result, although I now realized the ipa transcript still have mistakes in it. For instance, there is an ordinari ni used for に, there should be the IPA equivalent of ňi, as that's how the Japanese pronounce this.
The model learned to compenate for this, so when your input also have the same mistake, it speaks fine.

0 replies

neurlang · 2022-06-01T03:47:56Z

neurlang
Jun 1, 2022
Author

#月には叢雲　華には風と
#朧深に隠れた　焦燥

tts --text="tsuki ni vɑ murɑkumɔ hɑnɑ ni vɑ kɑzɛ tɔ. ɔbɔrɔmi ni kɑkurɛtɑ ʃɔusɔ." $enc --out_path tts-poem-1.wɑv
ffmpeg -y -i tts-poem-1.wɑv tts-poem-1.mp4

tts-poem-1.mp4

#瞼焼き付いた顔
#理解者の証さえ

tts --text="mɑbutɑ jɑkitsuitɑ kɑvɔ. rikɑiʃɑ nɔ ɑkɑʃi sɑɛ." $enc --out_path tts-poem-2.wɑv
ffmpeg -y -i tts-poem-2.wɑv tts-poem-2.mp4

tts-poem-2.mp4

#刹那、退屈の隙間贖い　心燻り
#不安を産み出した

tts --text="sɛtsunɑ, tɑikutsu nɔ sukimɑ ɑgɑnɑi kɔkɔrɔ kusuburi. huɑn vɔ umidɑʃitɑ." $enc --out_path tts-poem-3.wɑv
ffmpeg -y -i tts-poem-3.wɑv tts-poem-3.mp4

tts-poem-3.mp4

#盲目消えた安らぎに出会って
#芽生えた恋情　譲る気は無い

tts --text="mɔːmɔku kiɛtɑ jɑsurɑgi ni dɛɑttɛ. mɛbɑɛtɑ rɛndʒɔu juzuru ki vɑ nɑi." $enc --out_path tts-poem-4.wɑv
ffmpeg -y -i tts-poem-4.wɑv tts-poem-4.mp4

tts-poem-4.mp4

#月には叢雲　華には風と
#此方より彼方へ永久　築けぬなら
#雲突き抜け　風斬り裂いて
#久遠の揺蕩いへ　誘う

tts --text="tsuki ni vɑ murɑkumɔ hɑnɑ ni vɑ kɑzɛ tɔ. kɔnɑtɑ jɔri kɑnɑtɑ ɛ ɛikjuː kizukɛnu nɑrɑ." $enc --out_path tts-poem-5.wɑv
ffmpeg -y -i tts-poem-5.wɑv tts-poem-5.mp4

tts-poem-5.mp4

tts --text="kumɔ tsukinukɛ kɑzɛ kirisɑitɛ. kuɔn nɔ tɑjutɑi ɛ izɑnɑu." $enc --out_path tts-poem-6.wɑv
ffmpeg -y -i tts-poem-6.wɑv tts-poem-6.mp4

tts-poem-6.mp4

#指と指が絡まり
#孤独を枯らしてゆく

tts --text="jubi tɔ jubi gɑ kɑrɑmɑri. kɔdɔku ɔ kɑrɑʃita juku." $enc --out_path tts-poem-7.wɑv
ffmpeg -y -i tts-poem-7.wɑv tts-poem-7.mp4

tts-poem-7.mp4

#真摯ゆえの想いの偏りは
#互いの意思を　繋ぐ要にして

tts --text="ʃinʃi juɛ nɔ ɔmɔi nɔ kɑtɑjɔri vɑ. tɑgɑi nɔ iʃi ɔ tsunɑgu kɑnɑmɛ ni ʃitɛ." $enc --out_path tts-poem-8.wɑv
ffmpeg -y -i tts-poem-8.wɑv tts-poem-8.mp4

tts-poem-8.mp4

#毒にも似たこの絆は
#僅かな終焉の予感じゃ　千切れはしない

tts --text="dɔku ni mɔ nitɑ kɔnɔ kizunɑ vɑ. vɑzukɑ nɑ ʃuːɛn nɔ jɔkɑn dʒɑ tʃigirɛ vɑ ʃinɑi." $enc --out_path tts-poem-9.wɑv
ffmpeg -y -i tts-poem-9.wɑv tts-poem-9.mp4

tts-poem-9.mp4

#月には叢雲　華には風と
#たとえ愛の唄が永久　届かぬとも
#待ち続けよう　輪廻の時を
#慈しむ心に　誓って

tts --text="tsuki ni vɑ murɑkumɔ hɑnɑ ni vɑ kɑzɛ tɔ. tɑtɔɛ ɑi nɔ utɑ gɑ ɛikjuː tɔdɔkɑnu tɔmɔ." $enc --out_path tts-poem-10.wɑv
ffmpeg -y -i tts-poem-10.wɑv tts-poem-10.mp4

tts-poem-10.mp4

tts --text="mɑtʃi tsuzukɛjɔu rinnɛ nɔ tɔki vɔ. itsukuʃimu kɔkɔrɔ ni tʃikɑttɛ." $enc --out_path tts-poem-11.wɑv
ffmpeg -y -i tts-poem-11.wɑv tts-poem-11.mp4

tts-poem-11.mp4

#盲目消えた安らぎに出会って
#芽生えた恋情　譲る気は無い

tts --text="mɔːmɔku kiɛtɑ jɑsurɑgi ni dɛɑttɛ. mɛbɑɛtɑ rɛndʒɔu juzuru ki vɑ nɑi." $enc --out_path tts-poem-12.wɑv
ffmpeg -y -i tts-poem-12.wɑv tts-poem-12.mp4

tts-poem-12.mp4

#月には叢雲　華には風と
#此方より彼方へ永久　築けぬなら
#雲突き抜け　風斬り裂いて
#久遠の揺蕩いへ　誘う

tts --text="tsuki ni vɑ murɑkumɔ hɑnɑ ni vɑ kɑzɛ tɔ. kɔnɑtɑ jɔri kɑnɑtɑ ɛ ɛikjuː kizukɛnu nɑrɑ." $enc --out_path tts-poem-13.wɑv
ffmpeg -y -i tts-poem-13.wɑv tts-poem-13.mp4

tts-poem-13.mp4

tts --text="kumɔ tsukinukɛ kɑzɛ kirisɑitɛ. kuɔn nɔ tɑjutɑi ɛ izɑnɑu." $enc --out_path tts-poem-14.wɑv
ffmpeg -y -i tts-poem-14.wɑv tts-poem-14.mp4

tts-poem-14.mp4

#月には叢雲　華には風と
#たとえ愛の唄が永久　届かぬとも
#待ち続けよう　輪廻の時を
#慈しむ心に　誓って

tts --text="tsuki ni vɑ murɑkumɔ hɑnɑ ni vɑ kɑzɛ tɔ. tɑtɔɛ ɑi nɔ utɑ gɑ ɛikjuː tɔdɔkɑnu tɔmɔ." $enc --out_path tts-poem-15.wɑv
ffmpeg -y -i tts-poem-15.wɑv tts-poem-15.mp4

tts-poem-15.mp4

tts --text="mɑtʃi tsuzukɛjɔu rinnɛ nɔ tɔki ɔ. itsukuʃimu kɔkɔrɔ ni tʃikɑttɛ." $enc --out_path tts-poem-16.wɑv
ffmpeg -y -i tts-poem-16.wɑv tts-poem-16.mp4

tts-poem-16.mp4

2 replies

kksteini Jun 1, 2022

いい声だ。ほんとに五日前より今は方がいいです。
I assume basic 5000 means you have that many sound files to train on?

dmzubr Jul 21, 2022

Hi!

What was the step count of model that generate the samples above?

neurlang · 2022-06-01T18:48:36Z

neurlang
Jun 1, 2022
Author

yeah 5000 files, I forgot where I got the dataset from. I'm training on the nv3090ti branch. The most significant change is removal of this line, with the line I could not get the model to acceptably converge:

#self.nets.extend([DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods])

Here is my config:

audio_config = BaseAudioConfig(
    sample_rate=48000,
    fft_size=1000,
    win_length=1000,
    hop_length=250,
    num_mels=80,
    preemphasis=0.0,
    ref_level_db=20,
    log_func="np.log",
    do_trim_silence=True,
    trim_db=45,
    mel_fmin=0,
    mel_fmax=8000,
    spec_gain=1.0,
    signal_norm=False,
    do_amp_to_db_linear=False,
)

config = VitsConfig(
    audio=audio_config,
    run_name="vits_ljspeech",
    batch_size=16,
    eval_batch_size=16,
    batch_group_size=5,
    num_loader_workers=8,
    num_eval_loader_workers=8,
    run_eval=True,
    save_step=1000,
    test_delay_epochs=-1,
    epochs=1000,
    text_cleaner="english_cleaners",
    use_phonemes=True,
    phoneme_language="multi-lingual",
    phonemizer="ipa_phonemizer",
    phoneme_cache_path=os.path.join(output_path, "phoneme_cache"),
    compute_input_seq_cache=True,
    print_step=25,
    print_eval=True,
    mixed_precision=True,
    output_path=output_path,
    datasets=[dataset_config],
)

0 replies

michaellin99999 · 2022-06-07T00:50:35Z

michaellin99999
Jun 7, 2022

is this the kokoro dataset from kaggle?

1 reply

neurlang Sep 18, 2022
Author

jsut-label: context labels and pronunciation data for JSUT corpus.

LICENCE:
Text in corpus is licensed as follows. The text data were modified and pronunciation information is added.
basic5000
- wikipedia (https://ja.wikipedia.org/) CC-BY-SA 3.0
- TANAKA corpus (http://www.edrdg.org/wiki/index.php/Tanaka_Corpus) CC-BY 2.0
- JSUT (Japanese speech corpus of Saruwatari-lab., University of Tokyo) (https://sites.google.com/site/shinnosuketakamichi/publication/jsut). CC-BY-SA 4.0

neurlang · 2022-09-18T13:09:35Z

neurlang
Sep 18, 2022
Author

I've retrained the model on latest coqui tts:

#何を探してる？
tts/tts --text="nɑɲi vɔ sɑgɑʃi tɛru." $enc --out_path tts-poem-b-1.wav

tts-poem-b-1.wav.mp4

#まだ分からないけど
tts/tts --text="mɑdɑ vɑkɑrɑnɑikɛdɔ." $enc --out_path tts-poem-b-2.wav

tts-poem-b-2.wav.mp4

#入り口はここかも (Hand in hand)
tts/tts --text="irigutʃi vɑ kɔkɔ kɑ mɔ, hɑnd in hɑnd." $enc --out_path tts-poem-b-3.wav

tts-poem-b-3.wav.mp4

#それはキミの中に
tts/tts --text="sɔrɛ vɑ kimi nɔ nɑkɑ ɲi." $enc --out_path tts-poem-b-4.wav

tts-poem-b-4.wav.mp4

#眠ってる情熱もうすぐ目覚めそう
tts/tts --text="nɛmuttɛru dʒɔuɔnɛtsu mɔːuɔsugu mɛzɑmɛsɔː." $enc --out_path tts-poem-b-5.wav

tts-poem-b-5.wav.mp4

#気づいて！(Hand in Hand)
tts/tts --text="kidzuitɛ, hɑnd in hɑnd." $enc --out_path tts-poem-b-6.wav

tts-poem-b-6.wav.mp4

#夢が生まれる予感はただの
tts/tts --text="jumɛ gɑ humɑrɛru jɔkɑn vɑ tɑdɑ nɔ." $enc --out_path tts-poem-b-7.wav

tts-poem-b-7.wav.mp4

#錯覚じゃないはずさ (wow wo!)
tts/tts --text="sɑkkɑku dʒɑnɑi hɑzu sɑ, uɔu vɔ." $enc --out_path tts-poem-b-8.wav

tts-poem-b-8.wav.mp4

#動きださなきゃはじまらないよ
tts/tts --text="ugɔkidɑsɑnɑkjɑ hɑdʒimɑrɑnɑi jɔ." $enc --out_path tts-poem-b-9.wav

tts-poem-b-9.wav.mp4

#その手をつかまえたい
tts/tts --text="sɔnɔ tɛ vɔ tsukɑmɑɛtɑi." $enc --out_path tts-poem-b-10.wav

tts-poem-b-10.wav.mp4

#かわれかわれって今日から
tts/tts --text="kɑ vɑrɛ kɑ vɑrɛ ttɛ kjɔː kɑrɑ." $enc --out_path tts-poem-b-11.wav

tts-poem-b-11.wav.mp4

#新しい世界へと
tts/tts --text="ɑtɑrɑʃiː sɛkɑi ɛ tɔ." $enc --out_path tts-poem-b-12.wav

tts-poem-b-12.wav.mp4

#かわれかわれって元気に
tts/tts --text="kɑvɑ rɛ kɑvɑ rɛ ttɛ gɛnki ɲi." $enc --out_path tts-poem-b-13.wav

tts-poem-b-13.wav.mp4

#背中押してみよう
tts/tts --text="sɛnɑkɑɔ ʃitɛ mijɔu." $enc --out_path tts-poem-b-14.wav

tts-poem-b-14.wav.mp4

#いっしょにだったら飛べるかな
tts/tts --text="isʃɔ ɲidɑttɑrɑ tɔbɛru kɑ nɑ." $enc --out_path tts-poem-b-15.wav

tts-poem-b-15.wav.mp4

#じゃあためらわないでもう諦めたくない
tts/tts --text="dʒɑː tɑmɛrɑvɑnɑidɛ mɔː ɑkirɑmɛtɑkunɑi." $enc --out_path tts-poem-b-16.wav

tts-poem-b-16.wav.mp4

#決めたよ！
tts/tts --text="kimɛtɑ jɔ." $enc --out_path tts-poem-b-17.wav

tts-poem-b-17.wav.mp4

#これからさあどこへ行こう？
tts/tts --text="kɔrɛkɑrɑ sɑː dɔkɔ ɛ ikɔu." $enc --out_path tts-poem-b-18.wav

tts-poem-b-18.wav.mp4

#手に手をとって行こう！
tts/tts --text="tɛ ɲi tɛ ɔ tɔttɛ ikɔu." $enc --out_path tts-poem-b-19.wav

tts-poem-b-19.wav.mp4

#だれとめぐり逢う？
tts/tts --text="dɑrɛ tɔ mɛguri ɑu." $enc --out_path tts-poem-b-20.wav

tts-poem-b-20.wav.mp4

#いまがその時とピンときてないのかな？
tts/tts --text="imɑ gɑ sɔnɔtɔki tɔ pintɔ ki tɛnɑi nɔ kɑ nɑ." $enc --out_path tts-poem-b-21.wav

tts-poem-b-21.wav.mp4

#(Hand in hand)
tts/tts --text="hɑnd in hɑnd." $enc --out_path tts-poem-b-22.wav

tts-poem-b-22.wav.mp4

#直感を信じたいよ
tts/tts --text="tʃɔkkɑn ɔ ʃindʒitɑi jɔ." $enc --out_path tts-poem-b-23.wav

tts-poem-b-23.wav.mp4

#心がふるえて叫びたくなるんだ
tts/tts --text="kɔkɔrɔ gɑ furuɛtɛ sɑkɛbitɑku nɑru ndɑ." $enc --out_path tts-poem-b-24.wav

tts-poem-b-24.wav.mp4

#夢をかたちにしたくなったら
tts/tts --text="jumɛ ɔ kɑtɑtʃi ɲi ʃitɑku nɑttɑrɑ." $enc --out_path tts-poem-b-25.wav

tts-poem-b-25.wav.mp4

#ひとりきりじゃなくて (wow wo!)
tts/tts --text="hitɔri kiri dʒɑnɑkutɛ, uɔu vɔ." $enc --out_path tts-poem-b-26.wav

tts-poem-b-26.wav.mp4

#動き出したい同じ気持ちの
tts/tts --text="ugɔkidɑʃitɑi ɔnɑdʒi kimɔtʃi nɔ." $enc --out_path tts-poem-b-27.wav

tts-poem-b-27.wav.mp4

#仲間がほしくなる
tts/tts --text="nɑkɑmɑ gɑ hɔʃiku nɑru." $enc --out_path tts-poem-b-28.wav

tts-poem-b-28.wav.mp4

#おいでおいでって無茶は
tts/tts --text="ɔidɛ ɔidɛ ttɛ mutʃɑ vɑ." $enc --out_path tts-poem-b-29.wav

tts-poem-b-29.wav.mp4

#承知で誘っちゃうよ
tts/tts --text="ʃɔːtʃi dɛ sɑsɔttʃɑu jɔ." $enc --out_path tts-poem-b-30.wav

tts-poem-b-30.wav.mp4

#おいでおいでって勇気を
tts/tts --text="ɔidɛ ɔidɛ ttɛ juːki ɔ." $enc --out_path tts-poem-b-31.wav

tts-poem-b-31.wav.mp4

#だせばできるかも
tts/tts --text="dɑsɛbɑ dɛkiru kɑmɔ." $enc --out_path tts-poem-b-32.wav

tts-poem-b-32.wav.mp4

#一緒にどっかへ飛びたいよ
tts/tts --text="isʃɔ ɲi, dɔkkɑ ɛ, tɔbitɑi jɔ." $enc --out_path tts-poem-b-33.wav

tts-poem-b-33.wav.mp4

#じゃあその手ほら貸して
tts/tts --text="dʒɑː sɔnɔ tɛ, hɔrɑ kɑʃitɛ." $enc --out_path tts-poem-b-34.wav

tts-poem-b-34.wav.mp4

#ね行くんだよ！
tts/tts --text="nɛ iku ndɑ jɔ." $enc --out_path tts-poem-b-35.wav

tts-poem-b-35.wav.mp4

#夢が生まれる予感はただの
tts/tts --text="jumɛ gɑ umɑrɛru jɔkɑn vɑ tɑdɑ nɔ." $enc --out_path tts-poem-b-36.wav

tts-poem-b-36.wav.mp4

#錯覚じゃないはずさ wow wo!
tts/tts --text="sɑkkɑku dʒɑnɑi hɑzu sɑ uɔu vɔ." $enc --out_path tts-poem-b-37.wav

tts-poem-b-37.wav.mp4

#かわれかわれって今日から
tts/tts --text="kɑvɑ rɛ kɑvɑ rɛ ttɛ kjɔː kɑrɑ." $enc --out_path tts-poem-b-38.wav

tts-poem-b-38.wav.mp4

#新しい世界へと
tts/tts --text="ɑtɑrɑʃiː sɛkɑi ɛ tɔ." $enc --out_path tts-poem-b-39.wav

tts-poem-b-39.wav.mp4

#かわれかわれって元気に
tts/tts --text="kɑvɑ rɛ kɑvɑ rɛ ttɛ gɛnki ɲi." $enc --out_path tts-poem-b-40.wav

tts-poem-b-40.wav.mp4

#背中押してみよう
tts/tts --text="sɛnɑkɑɔ ʃitɛ mijɔu." $enc --out_path tts-poem-b-41.wav

tts-poem-b-41.wav.mp4

#おいでおいでって無茶は
tts/tts --text="ɔidɛ ɔidɛ ttɛ mutʃɑ vɑ." $enc --out_path tts-poem-b-42.wav

tts-poem-b-42.wav.mp4

#承知で誘っちゃうよ
tts/tts --text="ʃɔːtʃi dɛ sɑsɔttʃɑu jɔ." $enc --out_path tts-poem-b-43.wav

tts-poem-b-43.wav.mp4

#おいでおいでって勇気を
tts/tts --text="ɔidɛ ɔidɛ ttɛ juːki ɔ." $enc --out_path tts-poem-b-44.wav

tts-poem-b-44.wav.mp4

#だせばできるかも
tts/tts --text="dɑsɛbɑ dɛkiru kɑmɔ." $enc --out_path tts-poem-b-45.wav

tts-poem-b-45.wav.mp4

#いっしょにだったら飛べるかな
tts/tts --text="isʃɔ ɲidɑttɑrɑ tɔbɛru kɑ nɑ." $enc --out_path tts-poem-b-46.wav

tts-poem-b-46.wav.mp4

#じゃあためらわないでもう諦めたくない
tts/tts --text="dʒɑː tɑmɛrɑvɑnɑidɛ mɔː ɑkirɑmɛtɑkunɑi." $enc --out_path tts-poem-b-47.wav

tts-poem-b-47.wav.mp4

#決めたよ！
tts/tts --text="kimɛtɑ jɔ." $enc --out_path tts-poem-b-48.wav

tts-poem-b-48.wav.mp4

#これからさあどこへ行こう？
tts/tts --text="kɔrɛkɑrɑ sɑː dɔkɔ ɛ ikɔu." $enc --out_path tts-poem-b-49.wav

tts-poem-b-49.wav.mp4

#手に手をとって行こう！
tts/tts --text="tɛ ɲi tɛ ɔ tɔttɛ ikɔu." $enc --out_path tts-poem-b-50.wav

tts-poem-b-50.wav.mp4

10 replies

erogol Sep 21, 2022
Maintainer

There is one neural phonemizer here. https://github.com/as-ideas/DeepPhonemizer

If it works for you we can consider integrate it to 🐸TTS too

mthrok Sep 21, 2022

What are "マッチ試験" and "カットの新決議案" supposed to mean?

neurlang Sep 21, 2022
Author

There is one neural phonemizer here. https://github.com/as-ideas/DeepPhonemizer

If it works for you we can consider integrate it to frogTTS too

This is a good Idea. But to train the phonemizer, I don't have word-word pairs (only sentence-sentence currently).
It would take a lot of effort to align individual JP words to their ipa versions.

If anyone knows a neural phonemizer that can train on whole sentences that would be great.

neurlang Sep 21, 2022
Author

What are "マッチ試験" and "カットの新決議案" supposed to mean?

Even though this is my best model, I don't actually read japanese. Sorry. But it's from the dataset.

kdrkdrkdr Sep 30, 2022

Could you share your train_vits.py file?
It doesn't work with the options you posted.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Basic 5000] vits japanese model #1604

{{title}}

Replies: 6 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Basic 5000] vits japanese model #1604

neurlang May 27, 2022

Replies: 6 comments · 13 replies

erogol May 31, 2022 Maintainer

neurlang Jun 1, 2022 Author

neurlang Jun 1, 2022 Author

kksteini Jun 1, 2022

dmzubr Jul 21, 2022

neurlang Jun 1, 2022 Author

michaellin99999 Jun 7, 2022

neurlang Sep 18, 2022 Author

neurlang Sep 18, 2022 Author

erogol Sep 21, 2022 Maintainer

mthrok Sep 21, 2022

neurlang Sep 21, 2022 Author

neurlang Sep 21, 2022 Author

kdrkdrkdr Sep 30, 2022

neurlang
May 27, 2022

Replies: 6 comments 13 replies

erogol
May 31, 2022
Maintainer

neurlang
Jun 1, 2022
Author

neurlang
Jun 1, 2022
Author

neurlang
Jun 1, 2022
Author

michaellin99999
Jun 7, 2022

neurlang Sep 18, 2022
Author

neurlang
Sep 18, 2022
Author

erogol Sep 21, 2022
Maintainer

neurlang Sep 21, 2022
Author

neurlang Sep 21, 2022
Author