StarcoderPlus: RuntimeError: probability tensor contains either inf, nan or element < 0

[phalexo]

Should the same code "generate.py" be used for StarcoderPlus? Not very starry so far.

(StarCoder) developer@ai:~/starcoder/chat$ python generate.py --model_id /opt/data/data/bigcode/starcoderplus
Loading dialogue template from local directory
No dialogue template found in model repo. Defaulting to the no_system template.
=== SAMPLE PROMPT ===
<|system|>
<|end|>
<|user|>
Develop a C++ program that reads a text file line by line and counts the number of occurrences of a specific word in the file.<|end|>
<|assistant|>

Special tokens: {'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'additional_special_tokens': ['<|endoftext|>', '<fim_prefix>', '<fim_middle>', '<fim_suffix>', '<fim_pad>', '', '<gh_stars>', '<issue_start>', '<issue_comment>', '<issue_closed>', '<jupyter_start>', '<jupyter_text>', '<jupyter_code>', '<jupyter_output>', '<empty_output>', '<commit_before>', '<commit_msg>', '<commit_after>', '']}
EOS token ID for generation: 0

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/developer/mambaforge/envs/StarCoder did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/lib/openmpi'), PosixPath('/usr/local/cuda-11.8/lib'), PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nccl/lib'), PosixPath('/home/developer/anaconda3/lib')}
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11.8/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 5.2
CUDA SETUP: Detected CUDA version 118
/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Loading binary /home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so...
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████| 7/7 [01:52<00:00, 16.02s/it]
Traceback (most recent call last):
File "/home/developer/starcoder/chat/generate.py", line 139, in
main()
File "/home/developer/starcoder/chat/generate.py", line 119, in main
generated_ids = model.generate(**batch, generation_config=generation_config)
File "/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/transformers/generation/utils.py", line 1572, in generate
return self.sample(
File "/home/developer/mambaforge/envs/StarCoder/lib/python3.10/site-packages/transformers/generation/utils.py", line 2655, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either inf, nan or element < 0