Replies: 4 comments
-
|
>>> khalilrhouma |
Beta Was this translation helpful? Give feedback.
-
|
>>> khalilrhouma |
Beta Was this translation helpful? Give feedback.
-
|
>>> nmstoker |
Beta Was this translation helpful? Give feedback.
-
|
>>> khalilrhouma |
Beta Was this translation helpful? Give feedback.
-
>>> nmstoker
[August 18, 2020, 11:42pm]
,
I may have got mixed up (and if so, sorry for any wasting of time!)
but... slash
With Tacotron2, in the past I was sure that I'd been able to switch the
prenet setting during fine-tuning as per your comments
here, which indicate
applying a prenet of BN in the final stages to improve the spectrogram
quality. I'd tried this myself back in Feb/March time and got excellent
results.
However when I tried it with a recent run that's based on the dev brach
from a week ago, with the initial config.json file as per that below and
then trying to switch the prenet to BN, I got an error message as per
the end of the file training.1.log in the log details section below.
The command to do the fine-tuning was:
python mozilla_voice_tts/bin/train_tts.py --config_path /home/neil/main/Projects/TTSAug2020/models/neil17/neil17_G-August-15-2020_08+24PM-81dd362/config.json --restore_path /home/neil/main/Projects/TTSAug2020/models/neil17/neil17_G-August-15-2020_08+24PM-81dd362/checkpoint_320000.pth.tar
And I'd edited the value of prenet_type from 'original' to 'bn' in the
config.json within the directory supplied, so the command above would've
picked it up.
Q. Has something changed about how this is handled or have I maybe
got mixed up in what I recall changing?
Thanks, slash
Neil
## Details
### Platform OS
### Python Environment
### Package Installation
slash
Click to see package list (package count: 133)
slash
Package list from Conda
Package Version
------------------------ ---------------
slash _libgcc_mutex 0.1
slash openmp_mutex 4.5
absl-py 0.9.0
astroid 2.4.2
astunparse 1.6.3
attrdict 2.0.1
attrs 19.3.0
blas 1.0
bokeh 1.4.0
ca-certificates 2020.6.20
cachetools 4.1.1
cardboardlint 1.3.0
certifi 2020.6.20
cffi 1.14.1
chardet 3.0.4
click 7.1.2
clldutils 3.5.2
colorful 0.5.4
colorlog 4.2.1
csvw 1.7.0
cudatoolkit 10.2.89
cycler 0.10.0
decorator 4.4.2
filelock 3.0.12
flask 1.1.2
freetype 2.10.2
future 0.18.2
fuzzywuzzy 0.18.0
gast 0.3.3
google-auth 1.20.1
google-auth-oauthlib 0.4.1
google-pasta 0.2.0
grpcio 1.31.0
h5py 2.10.0
idna 2.10
importlib-metadata 1.7.0
inflect 4.1.0
intel-openmp 2020.1
isodate 0.6.0
isort 4.3.21
itsdangerous 2.0.0a1
jinja2 3.0.0a1
joblib 0.16.0
jpeg 9d
kenlm 0.0.0
keras-preprocessing 1.1.2
kiwisolver 1.2.0
lazy-object-proxy 1.4.3
lcms2 2.11
ld_impl_linux-64 2.34
libffi 3.2.1
libgcc-ng 9.3.0
libgomp 9.3.0
libpng 1.6.37
librosa 0.7.2
libstdcxx-ng 9.3.0
libtiff 4.1.0
libwebp-base 1.1.0
llvmlite 0.31.0
lz4-c 1.9.2
markdown 3.2.2
markupsafe 2.0.0a1
matplotlib 3.3.1
mccabe 0.6.1
mkl 2020.1
mkl-service 2.3.0
mkl_fft 1.1.0
mkl_random 1.1.1
mozilla-voice-tts 0.0.4+81dd362
ncurses 6.2
ninja 1.10.0
nose 1.3.7
numba 0.48.0
numpy 1.18.5
numpy-base 1.19.1
oauthlib 3.1.0
olefile 0.46
openssl 1.1.1g
opt-einsum 3.3.0
packaging 20.4
phonemizer 2.2.1
pillow 7.2.0
pip 20.2.2
protobuf 4.0.0rc2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.20
pylint 2.5.3
pyparsing 3.0.0a2
pysbd 0.3.1
pysocks 1.7.1
python 3.7.8
python-dateutil 2.8.1
python_abi 3.7
pytorch 1.6.0
pyyaml 5.3.1
readline 8.0
regex 2020.7.14
requests 2.24.0
requests-oauthlib 1.3.0
rfc3986 1.4.0
rsa 4.6
scikit-learn 0.23.2
scipy 1.4.1
segments 2.1.3
setuptools 49.6.0
six 1.15.0
soundfile 0.10.3.post1
sqlite 3.32.3
tabulate 0.8.7
tensorboard 2.3.0
tensorboard-plugin-wit 1.7.0
tensorboardx 2.1
tensorflow 2.3.0
tensorflow-estimator 2.3.0
termcolor 1.1.0
threadpoolctl 2.1.0
tk 8.6.10
toml 0.10.1
torchvision 0.7.0
tornado 6.0.4
tqdm 4.48.2
typed-ast 1.4.1
unidecode 0.04.20
uritemplate 3.0.1
urllib3 1.25.10
werkzeug 1.0.1
wheel 0.35.1
wrapt 1.12.1
xz 5.2.5
zipp 3.1.0
zlib 1.2.11
zstd 1.4.5
## Configuration
Click to see config file (lines: 163)
slash
Contents from: config_15Aug2020_A.json
{
'model': 'Tacotron2',
'run_name': 'neil17_G',
'run_description': 'tacotron2 older normalization with gradual training with cached heteronym phonemes',
// AUDIO PARAMETERS
'audio':{
// stft parameters
'fft_size': 1024, // number of stft frequency levels. Size of the linear spectogram frame.
'win_length': 1024, // stft window length in ms.
'hop_length': 256, // stft window hop-lengh in ms.
'frame_length_ms': null, // stft window length in ms.If null, 'win_length' is used.
'frame_shift_ms': null, // stft window hop-lengh in ms. If null, 'hop_length' is used.
// Audio processing parameters
'sample_rate': 22050, // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
'preemphasis': 0.99, // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
'ref_level_db': 0, // reference level db, theoretically 20db is the sound of air.
// Silence trimming
'do_trim_silence': true, // enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
'trim_db': 60, // threshold for timming silence. Set this according to your dataset.
// Griffin-Lim
'power': 1.8, // value to sharpen wav signals after GL algorithm.
'griffin_lim_iters': 60, // #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.
// MelSpectrogram parameters
'num_mels': 80, // size of the mel spec frame.
'mel_fmin': 40.0, // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
'mel_fmax': 8000.0, // maximum freq level for mel-spec. Tune for dataset!!
'spec_gain': 1.0, // scaler value appplied after log transform of spectrogram.
// Normalization parameters
'signal_norm': true, // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
'min_level_db': -10, // lower bound for normalization
'symmetric_norm': true, // move normalization to range [-1, 1]
'max_norm': 4.0, // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
'clip_norm': true, // clip normalized values into the range.
'stats_path': null // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
//'stats_path': '/home/neil/main/Projects/TTSJul2020/models/neil17/scale_stats.npy' // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},
// VOCABULARY PARAMETERS
// if custom character set is not defined,
// default set in symbols.py is used
//'characters':{
// 'pad': '',
// 'eos': '~',
// 'bos': '^',
// 'characters': 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!'(),-.:;? ',
// 'punctuations':'!'(),-.:;? ',
// 'phonemes':'ˈɪətnːdslɹkmðzɒˌæɛaʊɐwipevhfbɔʌʃuŋjɡɜɑʒθx'
//},
// DISTRIBUTED TRAINING
'distributed':{
'backend': 'nccl',
'url': 'tcp: slash / slash /localhost:54321'
},
'reinit_layers': [], // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.
// TRAINING
'batch_size': 32, // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
'eval_batch_size':16,
'r': 7, // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.
'gradual_training': [[0, 7, 64], [1, 5, 64], [50000, 3, 64], [130000, 2, 32], [290000, 1, 32]], //set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled. For Tacotron, you might need to reduce the 'batch_size' as you proceeed.
'loss_masking': true, // enable / disable loss masking against the sequence padding.
'ga_alpha': 10.0, // weight for guided attention loss. If > 0, guided attention is enabled.
'apex_amp_level': null, // level of optimization with NVIDIA's apex feature for automatic mixed FP16/FP32 precision (AMP), NOTE: currently only O1 is supported, and use 'O1' to activate.
// VALIDATION
'run_eval': true,
'test_delay_epochs': 12, //Until attention is aligned, testing only wastes computation time.
'test_sentences_file': null, // set a file to load sentences to be used for testing. If it is null then we use default english sentences.
// OPTIMIZER
'noam_schedule': false, // use noam warmup and lr schedule.
'grad_clip': 1.0, // upper limit for gradients for clipping.
'epochs': 1000, // total number of epochs to train.
'lr': 0.0001, // Initial learning rate. If Noam decay is active, maximum learning rate.
'wd': 0.000001, // Weight decay weight.
'warmup_steps': 4000, // Noam decay steps to increase the learning rate from 0 to 'lr'
'seq_len_norm': false, // Normalize eash sample loss with its length to alleviate imbalanced datasets. Use it if your dataset is small or has skewed distribution of sequence lengths.
// TACOTRON PRENET
'memory_size': -1, // ONLY TACOTRON - size of the memory queue used fro storing last decoder predictions for auto-regression. If < 0, memory queue is disabled and decoder only uses the last prediction frame.
'prenet_type': 'original', // 'original' or 'bn'.
'prenet_dropout': false, // enable/disable dropout at prenet.
// TACOTRON ATTENTION
'attention_type': 'original', // 'original' or 'graves'
'attention_heads': 4, // number of attention heads (only for 'graves')
'attention_norm': 'sigmoid', // softmax or sigmoid. Suggested to use softmax for Tacotron2 and sigmoid for Tacotron.
'windowing': false, // Enables attention windowing. Used only in eval mode.
'use_forward_attn': true, // if it uses forward attention. In general, it aligns faster.
'forward_attn_mask': false, // Additional masking forcing monotonicity only in eval mode.
'transition_agent': false, // enable/disable transition agent of forward attention.
'location_attn': true, // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
'bidirectional_decoder': false, // use https://arxiv.org/abs/1907.09006. Use it, if attention does not work well with your dataset.
'double_decoder_consistency': false, // use DDC explained here https://erogol.com/solving-attention-problems-of-tts-models-with-double-decoder-consistency-draft/
'ddc_r': 7, // reduction rate for coarse decoder.
// STOPNET
'stopnet': true, // Train stopnet predicting the end of synthesis.
'separate_stopnet': true, // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.
// TENSORBOARD and LOGGING
'print_step': 200, // Number of steps to log training on console.
'tb_plot_step': 100, // Number of steps to plot TB training figures.
'print_eval': false, // If True, it prints intermediate loss values in evalulation.
'save_step': 8000, // Number of training steps expected to save traninpg stats and checkpoints.
'checkpoint': true, // If true, it saves checkpoints per 'save_step'
'tb_model_param_stats': false, // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.
// DATA LOADING
'text_cleaner': 'phoneme_cleaners',
'enable_eos_bos_chars': false, // enable/disable beginning of sentence and end of sentence chars.
'num_loader_workers': 4, // number of training data loader processes. Don't set it too big. 4-8 are good values.
'num_val_loader_workers': 4, // number of evaluation data loader processes.
'batch_group_size': 0, //Number of batches to shuffle after bucketing.
'min_seq_len': 4, // DATASET-RELATED: minimum text length to use in training
'max_seq_len': 160, // DATASET-RELATED: maximum text length
// PATHS
'output_path': '/home/neil/main/Projects/TTSAug2020/models/neil17/',
// PHONEMES
'phoneme_cache_path': '/home/neil/main/Projects/TTSAug2020/models/neil17/neil17_uk_phonemes_full', // phoneme computation is slow, therefore, it caches results in the given folder.
'use_phonemes': true, // use phonemes instead of raw characters. It is suggested for better pronounciation.
'phoneme_language': 'en-gb-x-rp', // depending on your target language, pick one from https://github.com/bootphon/phonemizer#languages
// MULTI-SPEAKER and GST
'use_speaker_embedding': false, // use speaker embedding to enable multi-speaker learning.
'use_external_speaker_embedding_file': false, // if true, forces the model to use external embedding per sample instead of nn.embeddings, that is, it supports external embeddings such as those used at https://arxiv.org/abs /1806.04558
'external_speaker_embedding_file': '../../speakers-vctk-en.json', // if not null and use_external_speaker_embedding_file is true, it is used to load a specific embedding file and thus uses these embeddings instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
'style_wav_for_test': null, // path to style wav file to be used in TacotronGST inference.
'use_gst': false, // TACOTRON ONLY: use global style tokens
'gst': { // gst parameter if gst is enabled
'gst_style_input': null, // Condition the style input either on a
// -> wave file [path to wave] or
// -> dictionary using the style tokens {'token1': 'value', 'token2': 'value'} example {'0': 0.15, '1': 0.15, '5': -0.15}
// with the dictionary being len(dict) <= len(gst_style_tokens).
'gst_embedding_dim': 512,
'gst_num_heads': 4,
'gst_style_tokens': 10
},
// DATASETS
'datasets': // List of datasets. They all merged and they get different speaker_ids.
[
{
'name': 'ljspeech',
'path': '/home/neil/data/Projects/NeilTTS/neil16/',
'meta_file_train': 'metadata_train_n17.csv',
'meta_file_val': 'metadata_val_n17.csv'
}
]
}
## Logfile
Click to see log file (lines: 42)
slash
Logfile: training.1.log
2020-08-18 16:51:09.658786: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/opt/kaldi/tools/openfst/lib
2020-08-18 16:51:09.658809: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
> Using CUDA: True
> Number of GPUs: 1
> Git Hash: 81dd362
> Experiment folder: /home/neil/main/Projects/TTSAug2020/models/neil17/neil17_G-August-18-2020_04+51PM-81dd362
> Setting up Audio Processor...
| > sample_rate:22050
| > num_mels:80
| > min_level_db:-10
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:0
| > fft_size:1024
| > power:1.8
| > preemphasis:0.99
| > griffin_lim_iters:60
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:40.0
| > mel_fmax:8000.0
| > spec_gain:1.0
| > stft_pad_mode:reflect
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:60
| > do_sound_norm:False
| > stats_path:None
| > hop_length:256
| > win_length:1024
> Using model: Tacotron2
! Run is removed from /home/neil/main/Projects/TTSAug2020/models/neil17/neil17_G-August-18-2020_04+51PM-81dd362
Traceback (most recent call last):
File 'mozilla_voice_tts/bin/train_tts.py', line 714, in
main(args)
File 'mozilla_voice_tts/bin/train_tts.py', line 571, in main
model.load_state_dict(checkpoint['model'])
File '/home/neil/.conda/envs/tts_aug2020_2/lib/python3.7/site-packages/torch/nn/modules/module.py', line 1045, in load_state_dict
self.class.name, '
'.join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Tacotron2:
Missing key(s) in state_dict: 'decoder.prenet.linear_layers.0.batch_normalization.weight', 'decoder.prenet.linear_layers.0.batch_normalization.bias', 'decoder.prenet.linear_layers.0.batch_normalization.running_mean', 'decoder.prenet.linear_layers.0.batch_normalization.running_var', 'decoder.prenet.linear_layers.1.batch_normalization.weight', 'decoder.prenet.linear_layers.1.batch_normalization.bias', 'decoder.prenet.linear_layers.1.batch_normalization.running_mean', 'decoder.prenet.linear_layers.1.batch_normalization.running_var'.
- generated at 00:23 on Aug 19 2020 using
GatherUp tool
[This is an archived TTS discussion thread from discourse.mozilla.org/t/tacotron2-question-about-switching-to-bn-prenet]
Beta Was this translation helpful? Give feedback.
All reactions