Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

42 Open 229 Closed
42 Open 229 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

why use different judge model?
#716 opened Jan 9, 2025 by kydxh
@PhoenixZ810
1
MMMU-Pro?
#715 opened Jan 8, 2025 by yaogang2060
MMMU和MathVista在gpt-4o的结果
#712 opened Jan 6, 2025 by kydxh
1
ValueError: Image features and image tokens do not match
#710 opened Jan 4, 2025 by MoyusiteruIori
MMT-Bench Assertion Error
#709 opened Jan 4, 2025 by MoyusiteruIori
请问 CORE_MM benchmark 是否已经不支持了呢？
#707 opened Jan 4, 2025 by lihytotoro
本地评估模型相关疑问 awaiting confirm
#693 opened Dec 24, 2024 by Leke-G
@FangXinyu-0913
2
ChartQA/DocVQA 分数异常
#683 opened Dec 23, 2024 by OrlandoBloom16
@PhoenixZ810
1
MME-realworld benchmark如何设置local path？ awaiting confirm
#676 opened Dec 19, 2024 by ZihaoZheng98
2
AUTO_SPLIT=1 如何搭配CUDA_VISIBLE_DEVICES=使用
#674 opened Dec 19, 2024 by ZihaoZheng98
@FangXinyu-0913
2
The image quality in VLMEvalKit is compressed and lower than that provided by Benchmark itself.
#665 opened Dec 13, 2024 by Sync-yxh
1
Local judge LLM?
#664 opened Dec 12, 2024 by lyzhongcrd
3
AutoScoringJudge文件
#651 opened Dec 5, 2024 by wksuixin
@PhoenixZ810
1
LLAVA-one-vision MMMU 性能评测差别很大
#645 opened Dec 4, 2024 by linxid
2
Discrepancy in MM Math Evaluation Results: Possible Issue with Answer Extraction in VLMEvalkit
#638 opened Dec 1, 2024 by mantle2048
6
The evaluation device_map setting is not optimized for quantized models using the ExLlamaV2 backend.
#629 opened Nov 26, 2024 by WeiweiZhang1
Ovis1.5-Llama3-8B在Hallusion Bench上的指标和榜单上的指标差距过大
#595 opened Nov 13, 2024 by LIRENDA621
1
ChartQA eval bug
#584 opened Nov 7, 2024 by lemonliu1992
Corrupted MathVista testmini data WIP
#578 opened Nov 5, 2024 by LeoDu0314
1
AI2d gpt和claude3.5官方分数非常高 awaiting confirm
#577 opened Nov 5, 2024 by Violettttee
3
LLaVA v1.5 7b very low performance on TextVQA
#554 opened Oct 28, 2024 by wufeim
Different evaluate result between different VLMEvalKit version
#523 opened Oct 16, 2024 by terry-for-github
@kennymckormick
5
Are images compressed in tsv files?
#517 opened Oct 12, 2024 by ywwynm
3
[Help Wanted] the alignment with official accuracy in llama3.2-vision Feature Request Extra attention is needed
#493 opened Sep 29, 2024 by droidXrobot
8
Reproducing QWen2VL Results on Video Benchmarks with VLMEvalKit
#484 opened Sep 23, 2024 by aniki-ly
@FangXinyu-0913
4
ProTip! Add no:assignee to see everything that’s not assigned.