Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistent metric outputs #868

Closed
wants to merge 3 commits into from
Closed

Consistent metric outputs #868

wants to merge 3 commits into from

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Nov 18, 2020

To automate the use of metrics, they should return consistent outputs.
In particular I'm working on adding a conversion of metrics to keras metrics.
To achieve this we need two things:

  • have each metric return dictionaries of string -> floats since each keras metrics should return one float
  • define in the metric info the different fields of the output dictionary

In this PR I'm adding these two features.
I also fixed a few bugs in some metrics

#867 needs to be merged first

@lhoestq
Copy link
Member Author

lhoestq commented Nov 26, 2020

I keep this PR in stand-by for next week's datasets sprint. If the next release is 2.0.0 then we can include it given that it's breaking for many metrics

@albertvillanova
Copy link
Member

Metrics are deprecated in datasets and evaluate should be used instead: https://github.com/huggingface/evaluate

@albertvillanova albertvillanova deleted the consistent-metric-outputs branch September 24, 2023 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants