[Usage]: how to let vllm use the generation_config.json as the default generation config

[chiaitian]

Your current environment

The output of `python collect_env.py`

vLLM Version: 0.6.1.post2@9ba0817ff1eb514f51cc6de9cb8e16c98d6ee44f
model: Qwen2-VL-7B-Instruct

How would you like to use vllm

My starting cmd python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-VL-7B-Instruct --model /data/modelscope_cache/Qwen/Qwen2-VL-7B-Instruct

The log is showing

These configs are not the same as ones in the generation_config.json.

Two questions:

1. Does vllm use the generation_config.json as the default generation config?
2. If not, how to set?

[DarkLight1337]

Sorry for late reply. vLLM does use generation config, as shown in https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/config.py#L386

However I'm not sure whether this works for modelscope.

[hdnh2006]

Your current environment

The output of `python collect_env.py`

vLLM Version: 0.6.1.post2@9ba0817ff1eb514f51cc6de9cb8e16c98d6ee44f model: Qwen2-VL-7B-Instruct

How would you like to use vllm

My starting cmd python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-VL-7B-Instruct --model /data/modelscope_cache/Qwen/Qwen2-VL-7B-Instruct

The log is showing These configs are not the same as ones in the generation_config.json.

Two questions:

1. Does vllm use the generation_config.json as the default generation config?
2. If not, how to set?

I was facing the same issue and this is how I solved it: #11861 (comment)