Batch Inference Error for qwen2vl Model After Full Fine-Tuning

[Bravo5542]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.2.dev0
• Python version: 3.8.20
• PyTorch version: 2.4.1+cu121 (GPU)
• Transformers version: 4.46.1
• Datasets version: 3.1.0
• Accelerate version: 1.0.1
• PEFT version: 0.12.0
• TRL version: 0.9.6

Reproduction

I have fine-tuned the qwen2vl model using the command:

llamafactory-cli train examples/train_full/qwen2vl_full_sft.yaml

After saving the model in the "saves" directory, I attempted to perform batch inference using the provided script:

python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo

However, I encountered the following error:

ValueError: This model does not support image input.

1.The model_path I used points to the model saved after running the full fine-tuning script.
2.I have successfully used the LoRA fine-tuned model (trained with the lora_sft script and merged with merge_lora script), which allows for inference using the method provided in the qwen2vl documentation.
3.However, the model saved after full fine-tuning does not seem to support direct inference in the same way.

Others

No response

[hiyouga]

--template qwen2_vl

[Bravo5542]

thank u for your reply. However, i still want to know Why can't the full fine-tuned model be used in the same way as the original model or the model that has been fine-tuned with LoRA and merged?