Ks/refactor scoring endpoint #204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mschrimpf
reviewed
Sep 28, 2023
kvfairchild
reviewed
Nov 2, 2023
shehadak
force-pushed
the
ks/refactor_scoring_endpoint
branch
from
9736df3 to
1e28aef
Compare
shehadak
force-pushed
the
ks/refactor_scoring_endpoint
branch
2 times, most recently
from
f0098de to
46bf7fd
Compare
kvfairchild
reviewed
Nov 13, 2023
| @@ -41,11 +51,20 @@ def send_email_to_submitter(uid: int, domain: str, pr_number: str, | |||
|
|
|||
| if __name__ == '__main__': | |||
| parser = make_argparser() | |||
| parser.add_argument('--fn', type=str, nargs='?', default='run_scoring', | |||
There was a problem hiding this comment.
Is there a reason to add this here instead of including it directly in make_argparser()?
kvfairchild
reviewed
Nov 13, 2023
Comment on lines
36
to
43
| def resolve_models_benchmarks(args_dict: Dict[str, Union[str, List]]): | ||
| benchmarks, models = retrieve_models_and_benchmarks(args_dict) | ||
|
|
||
| run_scoring_endpoint(domain="language", jenkins_id=args_dict["jenkins_id"], | ||
| models=models, benchmarks=benchmarks, user_id=args_dict["user_id"], | ||
| model_type="artificialsubject", public=args_dict["public"], | ||
| competition=args_dict["competition"]) | ||
|
|
||
| model_ids = resolve_models(domain="language", models=models) | ||
| benchmark_ids = resolve_benchmarks(domain="language", benchmarks=benchmarks) | ||
| print("BS_NEW_MODELS=" + " ".join(model_ids)) | ||
| print("BS_NEW_BENCHMARKS=" + " ".join(benchmark_ids)) | ||
| return model_ids, benchmark_ids |
There was a problem hiding this comment.
Suggest moving resolve_models_benchmarks() to core
kvfairchild
approved these changes
Nov 13, 2023
shehadak
force-pushed
the
ks/refactor_scoring_endpoint
branch
from
1827097 to
67515e2
Compare
shehadak
force-pushed
the
ks/refactor_scoring_endpoint
branch
from
9425003 to
ba4fa4c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR separates the functionality of the language submission endpoint into two separate methods:
run_score, which calls the core scoring endpoint on every model/benchmark pair provided, andget_models_and_benchmarkswhich provides a list of model/benchmark pairs for scoring given a list of new models and new benchmarks. Previously, both functionalities were included in therun_scoremethod. However, motivated by the need to run scoring jobs on multiple nodes in an HPC instead of in a single job, this PR enables scripts to separately identify the list of model/benchmark pairs and run calls to the core scoring endpoint.This PR is complementary to brain-score/core#40
A note on backward compatibility: While this PR modifies
run_scoringinto two separate methods, the default functionality is still the same. In particular, if a user does not specify--fn=get_models_and_benchmarks, therun_scoringmethod will be called, which is the expected functionality prior to this PR.A note on
pyproject.toml: The commitaa21570is for debugging purposes and will be reverted before merging.