why use different judge model?

[kydxh]

I notice that in run.py, for different datasets, the judge models are different. For example, MCQ datasets use 'chatgpt-0125', and Mathvista uses 'gpt-4-turbo'. Why not use the same judge model for fair comparasion?

[PhoenixZ810]

Hi,

Thank you for your interest and feedback.

It's important to note that the judging model used for each benchmark is aligned with the configurations detailed in the original paper to ensure a fair and consistent comparison. Any changes to the judging model can lead to significantly different results.

We appreciate your understanding on this matter and are open to any further questions or discussions you might have.

[kydxh]

I got it. Thank you very much.
However, I still have some confusion. Do the choice of the judging model align with the dataset's paper or the model's paper? For different VLMs, is the judging model the same for the same dataset?
Additionally, does using different evaluation methods for different datasets raise concerns about fairness when comparing across datasets?

[PhoenixZ810]

Thank you for your interest and observations.

• The selection of the judging model strictly adheres to the configurations outlined in the original papers of the benchmarks.
• For various VLMs, we maintain consistency by using the same judging model when evaluating performance on the same benchmark.
• Different benchmarks indeed feature varied formats of questions and answers, such as multiple-choice QA and open-ended QA. As a result, the evaluation methods are tailored to fit the specific format of each benchmark. This will not lead to unfairness.

[kydxh]

I got it. Thank you very much.