AssertionError: Caught AssertionError in DataLoader worker process 0

[Lizh-TW]

I imitate recipes/kokoro/tacotron2-DDC to train a new chinese model.
but it can't train.
Error
run.sh: 3: run.sh: Bad substitution
/workspace/TTS/recipes/zh-cn/tacotron2-DDC

Setting up Audio Processor...
| > sample_rate:16000
| > resample:False
| > num_mels:80
| > min_level_db:-100
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:20
| > fft_size:1024
| > power:1.5
| > preemphasis:0.0
| > griffin_lim_iters:60
| > signal_norm:False
| > symmetric_norm:True
| > mel_fmin:50.0
| > mel_fmax:7600.0
| > spec_gain:1.0
| > stft_pad_mode:reflect
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:60
| > do_sound_norm:False
| > stats_path:None
| > base:10
| > hop_length:256
| > win_length:1024
There are 455 files.
100%|█████████████████████████████████████████████████████████████████████████████████████████| 455/455 [00:21<00:00, 20.99it/s]
Avg mel spec mean: -2.8795676691860432
Avg mel spec scale: 1.1610589694061608
Avg linear spec mean: -1.9986710445334874
Avg lienar spec scale: 1.1955652407877333
stats saved to /workspace/TTS/recipes/zh-cn/tacotron2-DDC/scale_stats.npy
Using CUDA: True
Number of GPUs: 1
Mixed precision mode is ON
Git Hash: d245b5d
Experiment folder: /workspace/TTS/recipes/zh-cn/tacotron2-DDC/cospro01_f01phr-June-25-2021_10+35PM-d245b5d4
Setting up Audio Processor...
| > sample_rate:16000
| > resample:False
| > num_mels:80
| > min_level_db:-100
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:20
| > fft_size:1024
| > power:1.5
| > preemphasis:0.0
| > griffin_lim_iters:60
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:50.0
| > mel_fmax:7600.0
| > spec_gain:1.0
| > stft_pad_mode:reflect
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:60
| > do_sound_norm:False
| > stats_path:/workspace/TTS/recipes/zh-cn/tacotron2-DDC/scale_stats.npy
| > base:10
| > hop_length:256
| > win_length:1024
| > Found 455 files in /workspace/TTS/recipes/zh-cn/tacotron2-DDC/cospro01_f01phr
Using model: Tacotron2

Model has 47915060 parameters
Starting with inf best loss.

DataLoader initialization
| > Use phonemes: True
| > phoneme language: zh-CN
| > Number of instances : 451
| > Max length sequence: 517
| > Min length sequence: 10
| > Avg length sequence: 227.52549889135256
| > Num. instances discarded by max-min (max=153, min=6) seq limits: 300
| > Batch group size: 4.

EPOCH: 0/1000

Number of output frames: 7

TRAINING (2021-06-25 22:35:29)
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 0.425 seconds.
Prefix dict has been built successfully.
Loading model cost 0.426 seconds.
Prefix dict has been built successfully.
Loading model cost 0.420 seconds.
Prefix dict has been built successfully.
! Run is removed from /workspace/TTS/recipes/zh-cn/tacotron2-DDC/cospro01_f01phr-June-25-2021_10+35PM-d245b5d4
Traceback (most recent call last):
File "/workspace/TTS/TTS/bin/train_tacotron.py", line 739, in
main(args)
File "/workspace/TTS/TTS/bin/train_tacotron.py", line 701, in main
train_avg_loss_dict, global_step = train(
File "/workspace/TTS/TTS/bin/train_tacotron.py", line 153, in train
for num_iter, data in enumerate(data_loader):
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/workspace/TTS/TTS/tts/datasets/TTSDataset.py", line 273, in getitem
return self.load_data(idx)
File "/workspace/TTS/TTS/tts/datasets/TTSDataset.py", line 166, in load_data
assert text.size > 0, self.items[idx][1]
AssertionError: /workspace/TTS/recipes/zh-cn/tacotron2-DDC/cospro01_f01phr/wav/cospro01_f01phr581.wav

Please tell me how to solve this problem, thank you.

[nicolalandro]

Can you show the text associated to the cospro01_f01phr581.wav file? It seams that the text is empty or maybe it become empty after preprocess.
(I start now to whatch this project I'm not an expert but that can be usefull for you)

[Lizh-TW]

metadata:
cospro01_f01phr581|看吧。|khˈɑnsi̪5 pˈɑwu2

preprocess:
def cospro01_f01phr(root_path, meta_file):
txt_file = os.path.join(root_path, meta_file)
items = []
speaker_name = "cospro01_f01phr"
with open(txt_file, "r") as ttf:
for line in ttf:
cols = line.split("|")
wav_file = os.path.join(root_path, "wav", cols[0] + ".wav")
text = cols[2].replace(" ", "")
items.append([text, wav_file, speaker_name])
return items

config:
{
"datasets": [
{
"name": "cospro01_f01phr",
"path": "DEFINE THIS",
"meta_file_train": "metadata.csv",
"meta_file_val": null
}
],
"audio": {
"fft_size": 1024,
"win_length": 1024,
"hop_length": 256,
"frame_length_ms": null,
"frame_shift_ms": null,
"sample_rate": 16000,
"preemphasis": 0.0,
"ref_level_db": 20,
"do_trim_silence": true,
"trim_db": 60,
"power": 1.5,
"griffin_lim_iters": 60,
"num_mels": 80,
"mel_fmin": 50.0,
"mel_fmax": 7600.0,
"spec_gain": 1,
"signal_norm": true,
"min_level_db": -100,
"symmetric_norm": true,
"max_norm": 4.0,
"clip_norm": true,
"stats_path": "scale_stats.npy"
},
"gst":{
"gst_style_input": null,

    "gst_embedding_dim": 512,
    "gst_num_heads": 4,
    "gst_style_tokens": 10,
    "gst_use_speaker_embedding": false
},
"model": "Tacotron2",
"run_name": "cospro01_f01phr",
"run_description": "tacotron2 with DDC and differential spectral loss.",
"batch_size": 32,
"eval_batch_size": 16,
"mixed_precision": true,
"distributed": {
    "backend": "nccl",
    "url": "tcp:\/\/localhost:54321"
},
"reinit_layers": [],
"loss_masking": true,
"decoder_loss_alpha": 0.5,
"postnet_loss_alpha": 0.25,
"postnet_diff_spec_alpha": 0.25,
"decoder_diff_spec_alpha": 0.25,
"decoder_ssim_alpha": 0.5,
"postnet_ssim_alpha": 0.25,
"ga_alpha": 5.0,
"stopnet_pos_weight": 15.0,
"run_eval": true,
"test_delay_epochs": 10,
"test_sentences_file": null,
"noam_schedule": false,
"grad_clip": 1.0,
"epochs": 4000,
"lr": 0.0001,
"wd": 0.000001,
"warmup_steps": 4000,
"seq_len_norm": false,
"memory_size": -1,
"prenet_type": "original",
"prenet_dropout": true,
"attention_type": "original",
"windowing": false,
"use_forward_attn": false,
"forward_attn_mask": false,
"transition_agent": false,
"location_attn": true,
"bidirectional_decoder": false,
"double_decoder_consistency": true,
"ddc_r": 7,
"attention_heads": 4,
"attention_norm": "sigmoid",
"r": 7,
"gradual_training": [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 32], [290000, 1, 32]],
"stopnet": true,
"separate_stopnet": true,
"print_step": 25,
"tb_plot_step": 100,
"print_eval": false,
"save_step": 10000,
"checkpoint": true,
"keep_all_best": false,
"keep_after": 10000,
"tb_model_param_stats": false,
"text_cleaner": "basic_cleaners",
"enable_eos_bos_chars": false,
"num_loader_workers": 4,
"num_val_loader_workers": 4,
"batch_group_size": 4,
"min_seq_len": 6,
"max_seq_len": 153,
"compute_input_seq_cache": false,
"use_noise_augment": true,
"output_path": "DEFINE THIS",
"phoneme_cache_path": "DEFINE THIS",
"use_phonemes": true,
"phoneme_language": "zh-CN",
"use_speaker_embedding": false,
"use_gst": false,       			
"use_external_speaker_embedding_file": false,
"external_speaker_embedding_file": "../../speakers-vctk-en.json"

}

[nicolalandro]

Under config>dataset you do not specify the Path. I read:
"path": "DEFINE THIS",

[Lizh-TW]

I specify it at step "training".

CUDA_VISIBLE_DEVICES="0" python TTS/bin/train_tacotron.py --config_path $RUN_DIR/tacotron2-DDC.json
--coqpit.output_path $RUN_DIR
--coqpit.datasets.0.path $RUN_DIR/$CORPUS
--coqpit.audio.stats_path $RUN_DIR/scale_stats.npy
--coqpit.phoneme_cache_path $RUN_DIR/phoneme_cache \