-
Notifications
You must be signed in to change notification settings - Fork 346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OpenAI output null values for keys are converted to True in validated response #479
Comments
Hi, thanks for reporting this issue, we'll look into it. |
My hunch here is that null is being read in as a Truthy string, and then getting translated to True. They should instead be dropped if the field is optional. if the field is mandatory, we should treat it as an error in skeletal validation. |
@irgolic , I'm following the progress. |
The offender appears to be here: By commenting out these lines, I am able to avoid this issue. That being said, I'm not totally sure what's going on here. @irgolic any ideas what this is doing? |
Hey all, any updates here? |
Hello @LakshmiPanguluri, we support multiple json suffixes that can be added to the prompt; those can be found here. Replacing the current Also, I noticed a few typos in the prompt. Here's a corrected version of the prompt (which also includes a recommended design according to multiple prompt designing guides available online):
(Basically, we replace the This is the raw and validated outputs I received after updating the prompt: @tbrownio Great catch! |
@tbrownio We looked into this in a bit more detail. When the field is optional and the value is Once we get that in e.g here and multiple other places, instead of returning the |
Thanks for following up here, and I'm looking forward to a fix.
Tyler Brown
***@***.***
Seattle, WA
https://tbrown.io ( https://tbrown.io/ )
https://twitter.com/tbrownio
https://linkedin.com/in/tbrownio ( https://www.linkedin.com/in/tbrownio/ )
…On Fri, Jan 12, 2024 at 9:45 AM, Karan Acharya < ***@***.*** > wrote:
@ tbrownio ( https://github.com/tbrownio ) We looked into this in a bit
more detail. When the field is optional and the value is null , we want to
exit earlier and hence we return True from the base class - Placeholder 's
verify method. Once we get that in e.g here (
https://github.com/guardrails-ai/guardrails/blob/8833c0507d58825b2712d7660b14efff7dc69855/guardrails/utils/json_utils.py#L129-L130
) and multiple other places, instead of returning the super_result directly
we should return None instead so as to avoid the main issue here. We're
planning to add this fix soon.
—
Reply to this email directly, view it on GitHub (
#479 (comment)
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AC3XTDDTX2JE3ESZJOGV72LYOFZDBAVCNFSM6AAAAABAC7QZISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBZG4YTENBZGE
).
You are receiving this because you were mentioned. Message ID: <guardrails-ai/guardrails/issues/479/1889712491
@ github. com>
|
A fix has been added for this issue! Merged in Closing this issue. @LakshmiPanguluri @tbrownio |
Describe the bug
OpenAI output null values for keys are converted to True in the validated response
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Library version:
Version (e.g. 0.1.5)
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: