DeepSpeed with Phi-3-mini-128K-instruct does not generate <|endoftext|> token

[shubhanshu786]

I am using Phi-3-mini-128K-instruct model for inference via DeepSpeed for high throughput and facing issue of model not generating <|endoftext|> during the generation time. For all my requests it generates all max_new_tokens size output and generation stops abruptly in the middle of the sentence due to token generation limit.

It generates <|end|> token in the text, which I am using as of now to limit the output but in some cases, it does not even generate <|end|> token as well.
Tested same model with vLLM engine and huggingface as well, and there it is able to generate <|endoftext|> token and stops token generation, so clearly issue is with DeepSpeed-MII only.

Expected Behaviour: During the inference <|endoftext|> (eos_token) should be generated and output generation should be stoped.

[2024-09-27 10:03:56,258] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja
ninja .................. [OKAY]

op name ................ installed .. compatible

async_io ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fp_quantizer ........... [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
/home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlopen' /home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlclose'
/home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlerror' /home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlsym'
collect2: error: ld returned 1 exit status
gds .................... [NO] ....... [NO]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.4
[WARNING] using untested triton version (3.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]

DeepSpeed general environment info:
torch install path ............... ['/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch']
torch version .................... 2.4.1+cu121
deepspeed install path ........... ['/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.15.1, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.4, cuda 12.1
shared memory (/dev/shm) size .... 7.72 GB

[loadams]

Thanks for reporting this @shubhanshu786 - I'll try to take a look.