Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve input mismatch error for inference requests (DLIS-6165) #330

Merged
merged 25 commits into from
Mar 9, 2024

Conversation

indrajit96
Copy link
Contributor

Created 2 new strings to improve the verbosity of the error response.

Scenario: When a inference request has less input params than the model expects error response is insufficient.
Fix: Added 2 new strings to the error response

Old Behavior:
[request id: <id_unknown>] expected 5 inputs but got 1 inputs for model 'llama2-7b'

New Behavior:
[request id: <id_unknown>] expected 5 inputs but got 1 inputs for model 'llama2-7b'. Got inputs ['prompt'], but missing ['max_output_token', 'top_k', 'top_p', 'temperature']"

We have added text to existing response so that existing tests do not break.
Working on the test case for this response will add later.

@indrajit96 indrajit96 requested a review from rmccorm4 February 28, 2024 00:06
@rmccorm4 rmccorm4 requested review from kthui and lkomali March 1, 2024 23:36
@rmccorm4 rmccorm4 changed the title DLIS-6165 Improve input mismatch error for inference requests Improve input mismatch error for inference requests (DLIS-6165) Mar 1, 2024
@lkomali
Copy link
Contributor

lkomali commented Mar 2, 2024

The message can be improved for better clarity and readability to something similar to the one below.

[request ID: <id_unknown>] Error: Expected 5 inputs for model 'llama2-7b', but received only 1 input(s).
Received inputs: ['prompt']
Missing inputs: ['max_output_token', 'top_k', 'top_p', 'temperature']
Please provide all required inputs.

@indrajit96
Copy link
Contributor Author

The message can be improved for better clarity and readability to something similar to the one below.

[request ID: <id_unknown>] Error: Expected 5 inputs for model 'llama2-7b', but received only 1 input(s). Received inputs: ['prompt'] Missing inputs: ['max_output_token', 'top_k', 'top_p', 'temperature'] Please provide all required inputs.

Fixed this made the message more verbose.

@kthui
Copy link
Contributor

kthui commented Mar 8, 2024

Please update the copyright year on both files, because this change is more than fixing a typo, i.e. on infer_request.h, the copyright should start with

// Copyright 2020-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

same for infer_request.cc

@indrajit96
Copy link
Contributor Author

Please update the copyright year on both files, because this change is more than fixing a typo, i.e. on infer_request.h, the copyright should start with

// Copyright 2020-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

same for infer_request.cc

Fixed for all files

Copy link
Contributor

@kthui kthui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@rmccorm4 rmccorm4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@indrajit96 indrajit96 merged commit 2bd46a4 into main Mar 9, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants