-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add palindrome_generation #39
feat: add palindrome_generation #39
Conversation
Can you please try to rebase to current main head? I first forgot to add the pytz dependency in pyproject.yaml. |
37f012c
to
d0d84ae
Compare
Hopefully did that correctly. If not I'll circle back on Sunday. |
OK, now formatting & unit tests pass. Since the result in not unique we need a score_answer() function which check whether the right letter-set was used and that the submitted answer is a palindrome. |
max_length: int = 10 seems like a low limit? Would love to see this done with 32K tokens, for instance.. |
@Miserlou I suggest we keep the default config very simple .. in ~1 week we'll have some experience of "curriculum engineering" and can then give some guidelines for "calibrating" the default/baseline complexity which can then be adjusted in a similar way for all tasks scalar value. |
I'm fine with changing the default config to whatever is reasonable. |
add palindrome score_answer & test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perfect! :-D I like the 0.0 - 0.05 reward leader .. keeps a tangible delta between right and wrong ;-)
# Check if the answer is a palindrome
if answer != answer[::-1]:
return 0.02
# Check if answer contains the same letters as provided (ignoring order)
if sorted(answer) != sorted(expected_letters):
return 0.05
Adds Palindrome Generation, to the Algorithmic section.
I made the decision to base this on made-up words. I think this will isolate the algorithmic challenge of creating a palindrome without interference from linguistic factors.
Prompt example:
Object example:
metadata: {'letters': ['c', 'c', 'f'], 'generated_palindrome': 'cfc'}
Can change min & max length as part of config.