Wml comp #1578
Merged
Wml comp #1578
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # examples/evaluate_image_text_to_text.py
# Conflicts: # src/unitxt/image_operators.py
# Conflicts: # prepare/augmentors/image_augmentors.py # src/unitxt/catalog/templates/qa/multiple_choice/with_context/lmms_eval.json # src/unitxt/catalog/templates/qa/with_context/lmms_eval.json
# Conflicts: # examples/evaluate_image_text_to_text.py # prepare/metrics/anls.py # src/unitxt/metrics.py # version.py
# Conflicts: # examples/evaluate_image_text_to_text.py # prepare/augmentors/image_augmentors.py # src/unitxt/augmentors.py # src/unitxt/inference.py
# Conflicts: # src/unitxt/catalog/templates/qa/multiple_choice/with_context/lmms_eval.json
# Conflicts: # examples/evaluate_image_text_to_text.py
…orrectness metric and with_type task
# Conflicts: # examples/evaluate_image_text_to_text.py # prepare/cards/ai2d.py # prepare/cards/chart_qa.py # prepare/cards/doc_vqa.py # prepare/cards/info_vqa.py # prepare/cards/websrc.py # prepare/metrics/exact_match_mm.py # prepare/metrics/relaxed_correctness.py # prepare/metrics/websrc_squad_f1.py # prepare/tasks/qa/tasks.py # prepare/templates/qa/with_context.py # src/unitxt/catalog/cards/ai2d.json # src/unitxt/catalog/cards/chart_qa_lmms_eval.json # src/unitxt/catalog/cards/doc_vqa/lmms_eval.json # src/unitxt/catalog/cards/info_vqa.json # src/unitxt/catalog/cards/websrc.json # src/unitxt/catalog/metrics/exact_match_mm.json # src/unitxt/catalog/metrics/relaxed_correctness.json # src/unitxt/catalog/metrics/websrc_squad_f1.json # src/unitxt/metrics.py
h '#' will be ignored, and an empty message aborts Merge
Signed-off-by: elronbandel <[email protected]>
Signed-off-by: elronbandel <[email protected]>
Signed-off-by: elronbandel <[email protected]>
… tests for set equality Signed-off-by: elronbandel <[email protected]>
…bles failure Signed-off-by: elronbandel <[email protected]>
elronbandel
approved these changes
Feb 6, 2025
dafnapension
pushed a commit
that referenced
this pull request
Feb 6, 2025
* minor bug fixes * adding anls * Added LMMS_eval template, exact_match_mm metric, RGB image augmentor. * Added qa.with_context_multiple choice lmms_eval template * Added relaxed_correctness metric, edited info_vqa card, edit lmms_eval template * minor script changes * minor run changes * Added default template * Added lmms-lab version of chartqa with default template and relaxed correctness metric and with_type task * Added lmms-lab card of info_vqa. added default template to info_vqa and docvqa * Wevsrc new with_domain task, new metric. WIP * websrc working * discarding not important changes from main * workaround for llama_vision with WML * WML vs local UNITXT reproductions * Added llama vision benchmark * Changes for commit * fix unitxt typo * ruff * Update inference engine model and adjust expected targets in tests Signed-off-by: elronbandel <[email protected]> * Fix WML Inference Engine tests for images Signed-off-by: elronbandel <[email protected]> * Update tests Signed-off-by: elronbandel <[email protected]> * Enhance CSV loader with low_memory option and update inference engine tests for set equality Signed-off-by: elronbandel <[email protected]> * Add error handling in Loader class to raise UnitxtError on load_iterables failure Signed-off-by: elronbandel <[email protected]> --------- Signed-off-by: elronbandel <[email protected]> Co-authored-by: Amit Alfassy [email protected] <[email protected]> Co-authored-by: Elron Bandel <[email protected]>
dafnapension
pushed a commit
that referenced
this pull request
Feb 10, 2025
* minor bug fixes * adding anls * Added LMMS_eval template, exact_match_mm metric, RGB image augmentor. * Added qa.with_context_multiple choice lmms_eval template * Added relaxed_correctness metric, edited info_vqa card, edit lmms_eval template * minor script changes * minor run changes * Added default template * Added lmms-lab version of chartqa with default template and relaxed correctness metric and with_type task * Added lmms-lab card of info_vqa. added default template to info_vqa and docvqa * Wevsrc new with_domain task, new metric. WIP * websrc working * discarding not important changes from main * workaround for llama_vision with WML * WML vs local UNITXT reproductions * Added llama vision benchmark * Changes for commit * fix unitxt typo * ruff * Update inference engine model and adjust expected targets in tests Signed-off-by: elronbandel <[email protected]> * Fix WML Inference Engine tests for images Signed-off-by: elronbandel <[email protected]> * Update tests Signed-off-by: elronbandel <[email protected]> * Enhance CSV loader with low_memory option and update inference engine tests for set equality Signed-off-by: elronbandel <[email protected]> * Add error handling in Loader class to raise UnitxtError on load_iterables failure Signed-off-by: elronbandel <[email protected]> --------- Signed-off-by: elronbandel <[email protected]> Co-authored-by: Amit Alfassy [email protected] <[email protected]> Co-authored-by: Elron Bandel <[email protected]>
dafnapension
pushed a commit
that referenced
this pull request
Feb 10, 2025
* minor bug fixes * adding anls * Added LMMS_eval template, exact_match_mm metric, RGB image augmentor. * Added qa.with_context_multiple choice lmms_eval template * Added relaxed_correctness metric, edited info_vqa card, edit lmms_eval template * minor script changes * minor run changes * Added default template * Added lmms-lab version of chartqa with default template and relaxed correctness metric and with_type task * Added lmms-lab card of info_vqa. added default template to info_vqa and docvqa * Wevsrc new with_domain task, new metric. WIP * websrc working * discarding not important changes from main * workaround for llama_vision with WML * WML vs local UNITXT reproductions * Added llama vision benchmark * Changes for commit * fix unitxt typo * ruff * Update inference engine model and adjust expected targets in tests Signed-off-by: elronbandel <[email protected]> * Fix WML Inference Engine tests for images Signed-off-by: elronbandel <[email protected]> * Update tests Signed-off-by: elronbandel <[email protected]> * Enhance CSV loader with low_memory option and update inference engine tests for set equality Signed-off-by: elronbandel <[email protected]> * Add error handling in Loader class to raise UnitxtError on load_iterables failure Signed-off-by: elronbandel <[email protected]> --------- Signed-off-by: elronbandel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes for bugs in running llama_vision using vllm and WML. New benchmark llama_vision which runs llama_vision_11B on AI2D, DocVQA, InfoVQA and ChartQa on WLM.