Allow to use local judge llm

[StarCycle]

Deploy a local language model as the judge / choice extractor

The default setting mentioned above uses OpenAI's GPT as the judge LLM. However, you can also deploy a local judge LLM with LMDeploy.

First install:

pip install lmdeploy openai

And then deploy a local judge LLM with the single line of code. LMDeploy will automatically download the model from Huggingface. Assuming we use internlm2-chat-1_8b as the judge, port 23333, and the key sk-123456 (the key must start with "sk-" and follow with any number you like):

lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333

You need to get the model name registered by LMDeploy with the following code:

from openai import OpenAI
client = OpenAI(
    api_key='sk-123456',
    base_url="http://0.0.0.0:23333/v1"
)
model_name = client.models.list().data[0].id

Now set some environment variables to tell VLMEvalKit how to use the local judge LLM. In fact, the local judge LLM mimics an online OpenAI model.

export OPENAI_API_KEY=sk-123456
export OPENAI_API_BASE=http://0.0.0.0:23333/v1/chat/completions
export LOCAL_LLM=<model_name you get>

Finally, you can run the commands in step 2 to evaluate your VLM with the local judge LLM.

Note that

• If you hope to deploy the judge LLM in a single GPU and evaluate your VLM on other GPUs because of limited GPU memory, try CUDA_VISIBLE_DEVICES=x like

CUDA_VISIBLE_DEVICES=0 lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
CUDA_VISIBLE_DEVICES=1,2,3 torchrun --nproc-per-node=3 run.py --data HallusionBench  --model qwen_chat --verbose

• If the local judge LLM is not good enough in following the instructions, the evaluation may fail. Please report such failures (e.g., by issues).
• It's possible to deploy the judge LLM in different ways, e.g., use a private LLM (not from HuggingFace) or use a quantized LLM. Please refer to the LMDeploy doc. You can use any other deployment framework if they support OpenAI API.