+
+
+
[discord/lmms-eval](https://discord.gg/zdkwKUqrPy)
+
+š [Supported Tasks (90+)](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/current_tasks.md) | š [Supported Models (30+)](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/models) | š [Documentation](docs/README.md)
+
+---
+
+## Annoucement
+- [2025-1] šš We introduce [VideoMMMU](https://videommmu.github.io/), a massive, multi-modal, multi-disciplinary video benchmark that evaluates the knowledge acquisition capability from educational videos.
+
+## Installation
+
+For formal usage, you can install the package from PyPI by running the following command:
+```bash
+pip install lmms-eval
+```
+
+For development, you can install the package by cloning the repository and running the following command:
+```bash
+git clone https://github.com/EvolvingLMMs-Lab/lmms-eval
+cd lmms-eval
+pip install -e .
+```
+
+If you want to test LLaVA, you will have to clone their repo from [LLaVA](https://github.com/haotian-liu/LLaVA) and
+```bash
+git clone https://github.com/LLaVA-VL/LLaVA-NeXT
+cd LLaVA-NeXT
+pip install -e .
+```
+
+## Evaluation
+
+**Evaluation of LLaVA-OneVision on VideoMMMU**
+
+```bash
+accelerate launch --num_processes=1 --main_process_port 12345 -m lmms_eval \
+--model llava_onevision \
+--model_args pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen,max_frames_num=32,torch_dype=bfloat16 \
+ --tasks video_mmmu \
+ --batch_size 1 \
+ --log_samples \
+ --log_samples_suffix debug \
+ --output_path ./logs/
+```
+
+**Evaluate a single track of VideoMMMU**
+
+```bash
+accelerate launch --num_processes=1 --main_process_port 12345 -m lmms_eval \
+--model llava_onevision \
+--model_args pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen,max_frames_num=32,torch_dype=bfloat16 \
+ --tasks video_mmmu_perception \
+ --batch_size 1 \
+ --log_samples \
+ --log_samples_suffix debug \
+ --output_path ./logs/
+```
+
+**Evaluate the question_only track of VideoMMMU (Knowledge Acquisition Experiment)**
+
+```bash
+accelerate launch --num_processes=1 --main_process_port 12345 -m lmms_eval \
+--model llava_onevision \
+--model_args pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen,max_frames_num=1,torch_dype=bfloat16 \
+ --tasks video_mmmu_adaptation_question_only \
+ --batch_size 1 \
+ --log_samples \
+ --log_samples_suffix debug \
+ --output_path ./logs/
+```
diff --git a/docs/README.md b/docs/README.md
new file mode 100755
index 0000000..2c91236
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,12 @@
+# LMMs Eval Documentation
+
+Welcome to the docs for `lmms-eval`!
+
+Majority of this documentation is adapted from [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/)
+
+## Table of Contents
+
+* To learn about the command line flags, see the [commands](commands.md)
+* To learn how to add a new moddel, see the [Model Guide](model_guide.md).
+* For a crash course on adding new tasks to the library, see our [Task Guide](task_guide.md).
+* If you need to upload your datasets into correct HF format with viewer supported, please refer to [tools](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/pufanyi/hf_dataset_docs/tools)
diff --git a/docs/commands.md b/docs/commands.md
new file mode 100755
index 0000000..3dd3427
--- /dev/null
+++ b/docs/commands.md
@@ -0,0 +1,231 @@
+# User Guide
+This document details the interface exposed by `lmms_eval` and provides details on what flags are available to users.
+
+## Command-line Interface
+
+
+Equivalently, running the library can be done via the `lmms_eval` entrypoint at the command line.
+
+This mode supports a number of command-line arguments, the details of which can be also be seen via running with `-h` or `--help`:
+
+- `--model` : Selects which model type or provider is evaluated. Must be a string corresponding to the name of the model type/provider being used. See [the main README](https://github.com/EleutherAI/lm-evaluation-harness/tree/main#model-apis-and-inference-servers) for a full list of enabled model names and supported libraries or APIs.
+
+* `--model_args` : Controls parameters passed to the model constructor. Accepts a string containing comma-separated keyword arguments to the model class of the format `"arg1=val1,arg2=val2,..."`, such as, for example `--model_args pretrained=liuhaotian/llava-v1.5-7b,batch_size=1`. For a full list of what keyword arguments, see the initialization of the corresponding model class in `lmms_eval/models/`.
+
+* `--tasks` : Determines which tasks or task groups are evaluated. Accepts a comma-separated list of task names or task group names. Must be solely comprised of valid tasks/groups. You can use `--tasks list` to see all the available tasks. If you add your own tasks but not shown on the list, you can try to set `--verbosity=DEBUG` to view the error message. You can also use `--tasks list_with_num` to check every tasks and the number of question each task contains. However, `list_with_num` will download all the available datasets and may require lots of memory and time.
+
+- `--num_fewshot` : Sets the number of few-shot examples to place in context. Must be an integer.
+
+- `--gen_kwargs` : takes an arg string in same format as `--model_args` and creates a dictionary of keyword arguments. These will be passed to the models for all called `generate_until` (free-form or greedy generation task) tasks, to set options such as the sampling temperature or `top_p` / `top_k`. For a list of what args are supported for each model type, reference the respective library's documentation (for example, the documentation for `transformers.AutoModelForCausalLM.generate()`.) These kwargs will be applied to all `generate_until` tasks called--we do not currently support unique gen_kwargs or batch_size values per task in a single run of the library. To control these on a per-task level, set them in that task's YAML file.
+
+- `--batch_size` : Sets the batch size used for evaluation. Can be a positive integer or `"auto"` to automatically select the largest batch size that will fit in memory, speeding up evaluation. One can pass `--batch_size auto:N` to re-select the maximum batch size `N` times during evaluation. This can help accelerate evaluation further, since `lm-eval` sorts documents in descending order of context length.
+
+- `--max_batch_size` : Sets the maximum batch size to try to fit in memory, if `--batch_size auto` is passed.
+
+- `--device` : Sets which device to place the model onto. Must be a string, for example, `"cuda", "cuda:0", "cpu", "mps"`. Defaults to "cuda", and can be ignored if running multi-GPU or running a non-local model type.
+
+- `--output_path` : A string of the form `dir/file.jsonl` or `dir/`. Provides a path where high-level results will be saved, either into the file named or into the directory named. If `--log_samples` is passed as well, then per-document outputs and metrics will be saved into the directory as well.
+
+- `--log_samples` : If this flag is passed, then the model's outputs, and the text fed into the model, will be saved at per-document granularity. Must be used with `--output_path`.
+
+- `--limit` : Accepts an integer, or a float between 0.0 and 1.0 . If passed, will limit the number of documents to evaluate to the first X documents (if an integer) per task or first X% of documents per task. Useful for debugging, especially on costly API models.
+
+- `--use_cache` : Should be a path where a sqlite db file can be written to. Takes a string of format `/path/to/sqlite_cache_` in order to create a cache db at `/path/to/sqlite_cache_rank{i}.db` for each process (0-NUM_GPUS). This allows results of prior runs to be cached, so that there is no need to re-run results in order to re-score or re-run a given (model, task) pair again.
+
+- `--cache_requests` : Can be "true", "refresh", or "delete". "true" means that the cache should be used. "refresh" means that you wish to regenerate the cache, which you should run if you change your dataset configuration for a given task. "delete" will delete the cache. Cached files are stored under lm_eval/cache/.cache unless you specify a different path via the environment variable: `LM_HARNESS_CACHE_PATH`. e.g. `LM_HARNESS_CACHE_PATH=~/Documents/cache_for_lm_harness`.
+
+- `--check_integrity` : If this flag is used, the library tests for each task selected are run to confirm task integrity.
+
+- `--write_out` : Used for diagnostic purposes to observe the format of task documents passed to a model. If this flag is used, then prints the prompt and gold target string for the first document of each task.
+
+- `--show_config` : If used, prints the full `lm_eval.api.task.TaskConfig` contents (non-default settings the task YAML file) for each task which was run, at the completion of an evaluation. Useful for when one is modifying a task's configuration YAML locally to transmit the exact configurations used for debugging or for reproducibility purposes.
+
+- `--include_path` : Accepts a path to a folder. If passed, then all YAML files containing `lm-eval` compatible task configurations will be added to the task registry as available tasks. Used for when one is writing config files for their own task in a folder other than `lm_eval/tasks/`.
+
+- `--system_instruction`: Specifies a system instruction string to prepend to the prompt.
+
+- `--apply_chat_template` : This flag specifies whether to apply a chat template to the prompt. It can be used in the following ways:
+ - `--apply_chat_template` : When used without an argument, applies the only available chat template to the prompt. For Hugging Face models, if no dedicated chat template exists, the default chat template will be applied.
+ - `--apply_chat_template template_name` : If the model has multiple chat templates, apply the specified template to the prompt.
+
+ For Hugging Face models, the default chat template can be found in the [`default_chat_template`](https://github.com/huggingface/transformers/blob/fc35907f95459d7a6c5281dfadd680b6f7b620e3/src/transformers/tokenization_utils_base.py#L1912) property of the Transformers Tokenizer.
+
+- `--fewshot_as_multiturn` : If this flag is on, the Fewshot examples are treated as a multi-turn conversation. Questions are provided as user content and answers are provided as assistant responses. Requires `--num_fewshot` to be set to be greater than 0, and `--apply_chat_template` to be on.
+
+- `--predict_only`: Generates the model outputs without computing metrics. Use with `--log_samples` to retrieve decoded results.
+
+* `--seed`: Set seed for python's random, numpy and torch. Accepts a comma-separated list of 3 values for python's random, numpy, and torch seeds, respectively, or a single integer to set the same seed for all three. The values are either an integer or 'None' to not set the seed. Default is `0,1234,1234` (for backward compatibility). E.g. `--seed 0,None,8` sets `random.seed(0)` and `torch.manual_seed(8)`. Here numpy's seed is not set since the second value is `None`. E.g, `--seed 42` sets all three seeds to 42.
+
+* `--wandb_args`: Tracks logging to Weights and Biases for evaluation runs and includes args passed to `wandb.init`, such as `project` and `job_type`. Full list [here](https://docs.wandb.ai/ref/python/init). e.g., ```--wandb_args project=test-project,name=test-run```
+
+* `--hf_hub_log_args` : Logs evaluation results to Hugging Face Hub. Accepts a string with the arguments separated by commas. Available arguments:
+ * `hub_results_org` - organization name on Hugging Face Hub, e.g., `EleutherAI`. If not provided, the results will be pushed to the owner of the Hugging Face token,
+ * `hub_repo_name` - repository name on Hugging Face Hub (deprecated, `details_repo_name` and `results_repo_name` should be used instead), e.g., `lm-eval-results`,
+ * `details_repo_name` - repository name on Hugging Face Hub to store details, e.g., `lm-eval-results`,
+ * `results_repo_name` - repository name on Hugging Face Hub to store results, e.g., `lm-eval-results`,
+ * `push_results_to_hub` - whether to push results to Hugging Face Hub, can be `True` or `False`,
+ * `push_samples_to_hub` - whether to push samples results to Hugging Face Hub, can be `True` or `False`. Requires `--log_samples` to be set,
+ * `public_repo` - whether the repository is public, can be `True` or `False`,
+ * `leaderboard_url` - URL to the leaderboard, e.g., `https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard`.
+ * `point_of_contact` - Point of contact for the results dataset, e.g., `yourname@example.com`.
+ * `gated` - whether to gate the details dataset, can be `True` or `False`.
+
+## External Library Usage
+
+We also support using the library's external API for use within model training loops or other scripts.
+
+`lmms_eval` supplies two functions for external import and use: `lmms_eval.evaluate()` and `lmms_eval.simple_evaluate()`.
+
+`simple_evaluate()` can be used by simply creating an `lmms_eval.api.model.LM` subclass that implements the methods described in the [Model Guide](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs/model_guide.md), and wrapping your custom model in that class as follows:
+
+```python
+import lmms_eval
+...
+
+my_model = initialize_my_model() # create your model (could be running finetuning with some custom modeling code)
+...
+# instantiate an LM subclass that takes your initialized model and can run
+# - `Your_LMM.loglikelihood()`
+# - `Your_LMM.generate_until()`
+lmm_obj = Your_LMM(model=my_model, batch_size=16)
+
+# indexes all tasks from the `lmms_eval/tasks` subdirectory.
+# Alternatively, you can set `TaskManager(include_path="path/to/my/custom/task/configs")`
+# to include a set of tasks in a separate directory.
+task_manager = lmms_eval.tasks.TaskManager()
+
+# Setting `task_manager` to the one above is optional and should generally be done
+# if you want to include tasks from paths other than ones in `lmms_eval/tasks`.
+# `simple_evaluate` will instantiate its own task_manager if it is set to None here.
+results = lmms_eval.simple_evaluate( # call simple_evaluate
+ model=lmm_obj,
+ tasks=["taskname1", "taskname2"],
+ num_fewshot=0,
+ task_manager=task_manager,
+ ...
+)
+```
+
+See the `simple_evaluate()` and `evaluate()` functions in [lmms_eval/evaluator.py](../lmms_eval/evaluator.py#:~:text=simple_evaluate) for a full description of all arguments available. All keyword arguments to simple_evaluate share the same role as the command-line flags described previously.
+
+Additionally, the `evaluate()` function offers the core evaluation functionality provided by the library, but without some of the special handling and simplification + abstraction provided by `simple_evaluate()`.
+
+As a brief example usage of `evaluate()`:
+
+```python
+import lmms_eval
+
+# suppose you've defined a custom lm_eval.api.Task subclass in your own external codebase
+from my_tasks import MyTask1
+...
+
+# create your model (could be running finetuning with some custom modeling code)
+my_model = initialize_my_model()
+...
+
+# instantiate an LM subclass that takes your initialized model and can run
+# - `Your_LM.loglikelihood()`
+# - `Your_LM.loglikelihood_rolling()`
+# - `Your_LM.generate_until()`
+lmm_obj = Your_LMM(model=my_model, batch_size=16)
+
+# optional: the task_manager indexes tasks including ones
+# specified by the user through `include_path`.
+task_manager = lmms_eval.tasks.TaskManager(
+ include_path="/path/to/custom/yaml"
+ )
+
+# To get a task dict for `evaluate`
+task_dict = lmms_eval.tasks.get_task_dict(
+ [
+ "mmlu", # A stock task
+ "my_custom_task", # A custom task
+ {
+ "task": ..., # A dict that configures a task
+ "doc_to_text": ...,
+ },
+ MyTask1 # A task object from `lm_eval.task.Task`
+ ],
+ task_manager # A task manager that allows lm_eval to
+ # load the task during evaluation.
+ # If none is provided, `get_task_dict`
+ # will instantiate one itself, but this
+ # only includes the stock tasks so users
+ # will need to set this if including
+ # custom paths is required.
+ )
+
+results = evaluate(
+ lm=lmm_obj,
+ task_dict=task_dict,
+ ...
+)
+```
+
+## Usage with SRT API
+
+> install sglang
+
+```bash
+git clone https://github.com/sgl-project/sglang.git
+cd sglang;
+pip install -e "python[srt]"
+python3 -m pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
+```
+
+> run sglang backend service with the following command
+
+```bash
+
+CKPT_PATH=$1
+TASK=$2
+MODALITY=$3
+TP_SIZE=$4
+echo $TASK
+TASK_SUFFIX="${TASK//,/_}"
+echo $TASK_SUFFIX
+
+python3 -m lmms_eval \
+ --model srt_api \
+ --model_args modality=$MODALITY,model_version=$CKPT_PATH,tp=$TP_SIZE,host=127.0.0.1,port=30000,timeout=600 \
+ --tasks $TASK \
+ --batch_size 1 \
+ --log_samples \
+ --log_samples_suffix $TASK_SUFFIX \
+ --output_path ./logs/
+
+```
+
+You may need to install some dependencies for the above command to work (if you encounter some errors).
+
+```bash
+pip install httpx==0.23.3
+pip install protobuf==3.20
+pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
+```
+
+
+
+## Regression Test
+
+Now after each PR, we need to run the regression test to make sure the performance of the model is not degraded.
+
+```bash
+python3 tools/regression.py
+```
+
+```bash
+Already on 'dev/fix_output_path'
+
+|task|llava-onevision-qwen2-0.5b-ov|
+|--|--|
+|ocrbench (dev/fix_output_path)|0.70 Ā± 0.70|
+|mmmu_val (dev/fix_output_path)|50.00 Ā± 50.00|
+|ai2d (dev/fix_output_path)|50.00 Ā± 50.00|
+|muirbench (dev/fix_output_path)|12.50 Ā± 12.50|
+|videomme (dev/fix_output_path)|2500.00 Ā± 2500.00|
+
+|branch|runtime|%|
+|--|--|--|
+|dev/fix_output_path|87.7s|100%|
+```
diff --git a/docs/current_tasks.md b/docs/current_tasks.md
new file mode 100644
index 0000000..8dc0a4a
--- /dev/null
+++ b/docs/current_tasks.md
@@ -0,0 +1,322 @@
+# Current Tasks
+
+> () indicates the task name in the lmms_eval. The task name is also used to specify the dataset in the configuration file.
+> The following is manually updated documentation. You could use `lmms_eval task --list` to list all supported tasks and their task names.
+
+## 1. Image tasks:
+
+- [AI2D](https://arxiv.org/abs/1603.07396) (ai2d)
+- [ChartQA](https://github.com/vis-nlp/ChartQA) (chartqa)
+- [COCO Caption](https://github.com/tylin/coco-caption) (coco_cap)
+ - COCO 2014 Caption (coco2014_cap)
+ - COCO 2014 Caption Validation (coco2014_cap_val)
+ - COCO 2014 Caption Test (coco2014_cap_test)
+ - COCO 2017 Caption (coco2017_cap)
+ - COCO 2017 Caption MiniVal (coco2017_cap_val)
+ - COCO 2017 Caption MiniTest (coco2017_cap_test)
+- [ConBench](https://github.com/foundation-multimodal-models/ConBench) (conbench)
+- [DetailCaps-4870](https://github.com/foundation-multimodal-models/CAPTURE) (detailcaps)
+- [DOCVQA](https://github.com/anisha2102/docvqa) (docvqa)
+ - DOCVQA Validation (docvqa_val)
+ - DOCVQA Test (docvqa_test)
+- [Ferret](https://github.com/apple/ml-ferret) (ferret)
+- [Flickr30K](https://github.com/BryanPlummer/flickr30k_entities) (flickr30k)
+ - Flickr30K Test (flickr30k_test)
+- [GQA](https://cs.stanford.edu/people/dorarad/gqa/index.html) (gqa)
+- [GQA-ru](https://huggingface.co/datasets/deepvk/GQA-ru) (gqa_ru)
+- [II-Bench](https://github.com/II-Bench/II-Bench) (ii_bench)
+- [Infographic VQA](https://www.docvqa.org/datasets/infographicvqa) (infovqa)
+ - Infographic VQA Validation (infovqa_val)
+ - Infographic VQA Test (infovqa_test)
+- [LiveBench](https://huggingface.co/datasets/lmms-lab/LiveBench) (live_bench)
+ - LiveBench 06/2024 (live_bench_2406)
+ - LiveBench 07/2024 (live_bench_2407)
+- [LLaVA-Bench-Wilder](https://huggingface.co/datasets/lmms-lab/LLaVA-Bench-Wilder) (llava_wilder_small)
+- [LLaVA-Bench-COCO](https://llava-vl.github.io/) (llava_bench_coco)
+- [LLaVA-Bench](https://llava-vl.github.io/) (llava_in_the_wild)
+- [MathVerse](https://github.com/ZrrSkywalker/MathVerse) (mathverse)
+ - MathVerse Text Dominant (mathverse_testmini_text_dominant)
+ - MathVerse Text Only (mathverse_testmini_text_only)
+ - MathVerse Text Lite (mathverse_testmini_text_lite)
+ - MathVerse Vision Dominant (mathverse_testmini_vision_dominant)
+ - MathVerse Vision Intensive (mathverse_testmini_vision_intensive)
+ - MathVerse Vision Only (mathverse_testmini_vision_only)
+- [MathVista](https://mathvista.github.io/) (mathvista)
+ - MathVista Validation (mathvista_testmini)
+ - MathVista Test (mathvista_test)
+- [MMBench](https://github.com/open-compass/MMBench) (mmbench)
+ - MMBench English (mmbench_en)
+ - MMBench English Dev (mmbench_en_dev)
+ - MMBench English Test (mmbench_en_test)
+ - MMBench Chinese (mmbench_cn)
+ - MMBench Chinese Dev (mmbench_cn_dev)
+ - MMBench Chinese Test (mmbench_cn_test)
+- [MME](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation) (mme)
+- [MMStar](https://github.com/MMStar-Benchmark/MMStar) (mmstar)
+- [MMUPD](https://huggingface.co/datasets/MM-UPD/MM-UPD) (mmupd)
+ - MMUPD Base (mmupd_base)
+ - MMAAD Base (mmaad_base)
+ - MMIASD Base (mmiasd_base)
+ - MMIVQD Base (mmivqd_base)
+ - MMUPD Option (mmupd_option)
+ - MMAAD Option (mmaad_option)
+ - MMIASD Option (mmiasd_option)
+ - MMIVQD Option (mmivqd_option)
+ - MMUPD Instruction (mmupd_instruction)
+ - MMAAD Instruction (mmaad_instruction)
+ - MMIASD Instruction (mmiasd_instruction)
+ - MMIVQD Instruction (mmivqd_instruction)
+- [MMVet](https://github.com/yuweihao/MM-Vet) (mmvet)
+- [Multilingual LlaVa Bench](https://huggingface.co/datasets/gagan3012/multilingual-llava-bench)
+ - llava_in_the_wild_arabic
+ - llava_in_the_wild_bengali
+ - llava_in_the_wild_chinese
+ - llava_in_the_wild_french
+ - llava_in_the_wild_hindi
+ - llava_in_the_wild_japanese
+ - llava_in_the_wild_russian
+ - llava_in_the_wild_spanish
+ - llava_in_the_wild_urdu
+- [NaturalBench](https://huggingface.co/datasets/BaiqiL/NaturalBench)
+- [NoCaps](https://nocaps.org/) (nocaps)
+ - NoCaps Validation (nocaps_val)
+ - NoCaps Test (nocaps_test)
+- [OCRBench](https://github.com/Yuliang-Liu/MultimodalOCR) (ocrbench)
+- [OKVQA](https://okvqa.allenai.org/) (ok_vqa)
+ - OKVQA Validation 2014 (ok_vqa_val2014)
+- [POPE](https://github.com/RUCAIBox/POPE) (pope)
+- [RefCOCO](https://github.com/lichengunc/refer) (refcoco)
+ - refcoco_seg_test
+ - refcoco_seg_val
+ - refcoco_seg_testA
+ - refcoco_seg_testB
+ - refcoco_bbox_test
+ - refcoco_bbox_val
+ - refcoco_bbox_testA
+ - refcoco_bbox_testB
+- [RefCOCO+](https://github.com/lichengunc/refer) (refcoco+)
+ - refcoco+\_seg
+ - refcoco+\_seg_val
+ - refcoco+\_seg_testA
+ - refcoco+\_seg_testB
+ - refcoco+\_bbox
+ - refcoco+\_bbox_val
+ - refcoco+\_bbox_testA
+ - refcoco+\_bbox_testB
+- [RefCOCOg](https://github.com/lichengunc/refer) (refcocog)
+ - refcocog_seg_test
+ - refcocog_seg_val
+ - refcocog_bbox_test
+ - refcocog_bbox_val
+- [ScienceQA](https://scienceqa.github.io/) (scienceqa_full)
+ - ScienceQA Full (scienceqa)
+ - ScienceQA IMG (scienceqa_img)
+- [ScreenSpot](https://github.com/njucckevin/SeeClick) (screenspot)
+ - ScreenSpot REC / Grounding (screenspot_rec)
+ - ScreenSpot REG / Instruction Generation (screenspot_reg)
+- [ST-VQA](https://rrc.cvc.uab.es/?ch=11) (stvqa)
+- [synthdog](https://github.com/clovaai/donut) (synthdog)
+ - synthdog English (synthdog_en)
+ - synthdog Chinese (synthdog_zh)
+- [TextCaps](https://textvqa.org/textcaps/) (textcaps)
+ - TextCaps Validation (textcaps_val)
+ - TextCaps Test (textcaps_test)
+- [TextVQA](https://textvqa.org/) (textvqa)
+ - TextVQA Validation (textvqa_val)
+ - TextVQA Test (textvqa_test)
+- [VCR-Wiki](https://github.com/tianyu-z/VCR)
+ - VCR-Wiki English
+ - VCR-Wiki English easy 100 (vcr_wiki_en_easy_100)
+ - VCR-Wiki English easy 500 (vcr_wiki_en_easy_500)
+ - VCR-Wiki English easy (vcr_wiki_en_easy)
+ - VCR-Wiki English hard 100 (vcr_wiki_en_hard_100)
+ - VCR-Wiki English hard 500 (vcr_wiki_en_hard_500)
+ - VCR-Wiki English hard (vcr_wiki_en_hard)
+ - VCR-Wiki Chinese
+ - VCR-Wiki Chinese easy 100 (vcr_wiki_zh_easy_100)
+ - VCR-Wiki Chinese easy 500 (vcr_wiki_zh_easy_500)
+ - VCR-Wiki Chinese easy (vcr_wiki_zh_easy)
+ - VCR-Wiki Chinese hard 100 (vcr_wiki_zh_hard_100)
+ - VCR-Wiki Chinese hard 500 (vcr_wiki_zh_hard_500)
+ - VCR-Wiki Chinese hard (vcr_wiki_zh_hard)
+- [VibeEval](https://github.com/reka-ai/reka-vibe-eval) (vibe_eval)
+- [VizWizVQA](https://vizwiz.org/tasks-and-datasets/vqa/) (vizwiz_vqa)
+ - VizWizVQA Validation (vizwiz_vqa_val)
+ - VizWizVQA Test (vizwiz_vqa_test)
+- [VQAv2](https://visualqa.org/) (vqav2)
+ - VQAv2 Validation (vqav2_val)
+ - VQAv2 Test (vqav2_test)
+- [WebSRC](https://x-lance.github.io/WebSRC/) (websrc)
+ - WebSRC Validation (websrc_val)
+ - WebSRC Test (websrc_test)
+- [WildVision-Bench](https://github.com/WildVision-AI/WildVision-Bench) (wildvision)
+ - WildVision 0617(wildvision_0617)
+ - WildVision 0630 (wildvision_0630)
+- [SeedBench 2 Plus](https://huggingface.co/datasets/AILab-CVC/SEED-Bench-2-plus) (seedbench_2_plus)
+
+## 2. Multi-image tasks:
+
+- [CMMMU](https://cmmmu-benchmark.github.io/) (cmmmu)
+ - CMMMU Validation (cmmmu_val)
+ - CMMMU Test (cmmmu_test)
+- [HallusionBench](https://github.com/tianyi-lab/HallusionBench) (hallusion_bench_image)
+- [ICON-QA](https://iconqa.github.io/) (iconqa)
+ - ICON-QA Validation (iconqa_val)
+ - ICON-QA Test (iconqa_test)
+- [JMMMU](https://mmmu-japanese-benchmark.github.io/JMMMU/) (jmmmu)
+- [LLaVA-NeXT-Interleave-Bench](https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-Interleave-Bench) (llava_interleave_bench)
+ - llava_interleave_bench_in_domain
+ - llava_interleave_bench_out_domain
+ - llava_interleave_bench_multi_view
+- [MIRB](https://github.com/ys-zong/MIRB) (mirb)
+- [MMMU](https://mmmu-benchmark.github.io/) (mmmu)
+ - MMMU Validation (mmmu_val)
+ - MMMU Test (mmmu_test)
+- [MMMU_Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro)
+ - MMMU Pro (mmmu_pro)
+ - MMMU Pro Original (mmmu_pro_original)
+ - MMMU Pro Vision (mmmu_pro_vision)
+ - MMMU Pro COT (mmmu_pro_cot)
+ - MMMU Pro Original COT (mmmu_pro_original_cot)
+ - MMMU Pro Vision COT (mmmu_pro_vision_cot)
+ - MMMU Pro Composite COT (mmmu_pro_composite_cot)
+- [MMT Multiple Image](https://mmt-bench.github.io/) (mmt_mi)
+ - MMT Multiple Image Validation (mmt_mi_val)
+ - MMT Multiple Image Test (mmt_mi_test)
+- [MuirBench](https://muirbench.github.io/) (muirbench)
+- [MP-DocVQA](https://github.com/rubenpt91/MP-DocVQA-Framework) (multidocvqa)
+ - MP-DocVQA Validation (multidocvqa_val)
+ - MP-DocVQA Test (multidocvqa_test)
+- [OlympiadBench](https://github.com/OpenBMB/OlympiadBench) (olympiadbench)
+ - OlympiadBench Test English (olympiadbench_test_en)
+ - OlympiadBench Test Chinese (olympiadbench_test_cn)
+- [Q-Bench](https://q-future.github.io/Q-Bench/) (qbenchs_dev)
+ - Q-Bench2-HF (qbench2_dev)
+ - Q-Bench-HF (qbench_dev)
+ - A-Bench-HF (abench_dev)
+
+## 3. Videos tasks:
+
+- [ActivityNet-QA](https://github.com/MILVLG/activitynet-qa) (activitynetqa_generation)
+- [SeedBench](https://github.com/AILab-CVC/SEED-Bench) (seedbench)
+- [SeedBench 2](https://github.com/AILab-CVC/SEED-Bench) (seedbench_2)
+- [CVRR-ES](https://github.com/mbzuai-oryx/CVRR-Evaluation-Suite) (cvrr)
+ - cvrr_continuity_and_object_instance_count
+ - cvrr_fine_grained_action_understanding
+ - cvrr_interpretation_of_social_context
+ - cvrr_interpretation_of_visual_context
+ - cvrr_multiple_actions_in_a_single_video
+ - cvrr_non_existent_actions_with_existent_scene_depictions
+ - cvrr_non_existent_actions_with_non_existent_scene_depictions
+ - cvrr_partial_actions
+ - cvrr_time_order_understanding
+ - cvrr_understanding_emotional_context
+ - cvrr_unusual_and_physically_anomalous_activities
+- [EgoSchema](https://github.com/egoschema/EgoSchema) (egoschema)
+ - egoschema_mcppl
+ - egoschema_subset_mcppl
+ - egoschema_subset
+- [LongVideoBench](https://github.com/longvideobench/LongVideoBench)
+- [MovieChat](https://github.com/rese1f/MovieChat) (moviechat)
+ - Global Mode for entire video (moviechat_global)
+ - Breakpoint Mode for specific moments (moviechat_breakpoint)
+- [MLVU](https://github.com/JUNJIE99/MLVU) (mlvu)
+- [MMT-Bench](https://mmt-bench.github.io/) (mmt)
+ - MMT Validation (mmt_val)
+ - MMT Test (mmt_test)
+- [MVBench](https://github.com/OpenGVLab/Ask-Anything/blob/main/video_chat2/MVBENCH.md) (mvbench)
+
+ - mvbench_action_sequence
+ - mvbench_moving_count
+ - mvbench_action_prediction
+ - mvbench_episodic_reasoning
+ - mvbench_action_antonym
+ - mvbench_action_count
+ - mvbench_scene_transition
+ - mvbench_object_shuffle
+ - mvbench_object_existence
+ - mvbench_fine_grained_pose
+ - mvbench_unexpected_action
+ - mvbench_moving_direction
+ - mvbench_state_change
+ - mvbench_object_interaction
+ - mvbench_character_order
+ - mvbench_action_localization
+ - mvbench_counterfactual_inference
+ - mvbench_fine_grained_action
+ - mvbench_moving_attribute
+ - mvbench_egocentric_navigation
+
+- [NExT-QA](https://github.com/doc-doc/NExT-QA) (nextqa)
+
+ - NExT-QA Multiple Choice Test (nextqa_mc_test)
+ - NExT-QA Open Ended Validation (nextqa_oe_val)
+ - NExT-QA Open Ended Test (nextqa_oe_test)
+
+- [PerceptionTest](https://github.com/google-deepmind/perception_test)
+
+ - PerceptionTest Test
+ - perceptiontest_test_mc
+ - perceptiontest_test_mcppl
+ - PerceptionTest Validation
+ - perceptiontest_val_mc
+ - perceptiontest_val_mcppl
+
+- [TempCompass](https://github.com/llyx97/TempCompass) (tempcompass)
+
+ - tempcompass_multi_choice
+ - tempcompass_yes_no
+ - tempcompass_caption_matching
+ - tempcompass_captioning
+
+
+- [TemporalBench](https://huggingface.co/datasets/microsoft/TemporalBench) (temporalbench)
+
+ - temporalbench_short_qa
+ - temporalbench_long_qa
+ - temporalbench_short_caption
+
+
+- [Vatex](https://eric-xw.github.io/vatex-website/index.html) (vatex)
+
+ - Vatex Chinese (vatex_val_zh)
+ - Vatex Test (vatex_test)
+
+- [VideoDetailDescription](https://huggingface.co/datasets/lmms-lab/VideoDetailCaption) (video_dc499)
+- [Video-ChatGPT](https://github.com/mbzuai-oryx/Video-ChatGPT) (videochatgpt)
+ - Video-ChatGPT Generic (videochatgpt_gen)
+ - Video-ChatGPT Temporal (videochatgpt_temporal)
+ - Video-ChatGPT Consistency (videochatgpt_consistency)
+- [Video-MME](https://video-mme.github.io/) (videomme)
+- [Vinoground](https://vinoground.github.io) (vinoground)
+- [VITATECS](https://github.com/lscpku/VITATECS) (vitatecs)
+
+ - VITATECS Direction (vitatecs_direction)
+ - VITATECS Intensity (vitatecs_intensity)
+ - VITATECS Sequence (vitatecs_sequence)
+ - VITATECS Compositionality (vitatecs_compositionality)
+ - VITATECS Localization (vitatecs_localization)
+ - VITATECS Type (vitatecs_type)
+
+- [WorldQA](https://zhangyuanhan-ai.github.io/WorldQA/) (worldqa)
+
+ - WorldQA Generation (worldqa_gen)
+ - WorldQA Multiple Choice (worldqa_mc)
+
+- [YouCook2](http://youcook2.eecs.umich.edu/) (youcook2_val)
+
+- [VDC](https://github.com/rese1f/aurora) (vdc)
+ - VDC Detailed Caption (detailed_test)
+ - VDC Camera Caption (camera_test)
+ - VDC Short Caption (short_test)
+ - VDC Background Caption (background_test)
+ - VDC Main Object Caption (main_object_test)
+
+
+## 4. Text Tasks
+
+- [GSM8K](https://github.com/openai/grade-school-math) (gsm8k)
+- [HellaSwag](https://rowanzellers.com/hellaswag/) (hellaswag)
+- [IFEval](https://github.com/google-research/google-research/tree/master/instruction_following_eval) (ifeval)
+- [MMLU](https://github.com/hendrycks/test) (mmlu)
+- [MMLU_pro](https://github.com/TIGER-AI-Lab/MMLU-Pro) (mmlu_pro)
diff --git a/docs/model_guide.md b/docs/model_guide.md
new file mode 100755
index 0000000..bc95509
--- /dev/null
+++ b/docs/model_guide.md
@@ -0,0 +1,78 @@
+# New Model Guide
+In order to properly evaluate a given LM, we require implementation of a wrapper class subclassing the `lmms_eval.api.model.lmms` class, that defines how the lmms_eval should interface with your model. This guide walks through how to write this `lmms` subclass via adding it to the library!
+
+## Setup
+
+To get started contributing, go ahead and fork the main repo, clone it, create a branch with the name of your task, and install the project requirements in your environment:
+
+```sh
+# After forking...
+git clone https://github.com/