Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add palindrome_generation #39

Merged
merged 3 commits into from
Feb 3, 2025

Conversation

joenorton
Copy link
Contributor

@joenorton joenorton commented Feb 1, 2025

Adds Palindrome Generation, to the Algorithmic section.
I made the decision to base this on made-up words. I think this will isolate the algorithmic challenge of creating a palindrome without interference from linguistic factors.

Prompt example:

q="Rearrange these letters to form a palindrome (a word, phrase, or sequence that remains the same in reverse): c, c, f

Example format:
racecar

What is your palindrome?", a="cfc"

Object example:
metadata: {'letters': ['c', 'c', 'f'], 'generated_palindrome': 'cfc'}

Can change min & max length as part of config.

@andreaskoepf
Copy link
Contributor

Can you please try to rebase to current main head? I first forgot to add the pytz dependency in pyproject.yaml.

@joenorton joenorton force-pushed the palindrome_generation branch from 37f012c to d0d84ae Compare February 1, 2025 02:46
@joenorton
Copy link
Contributor Author

Hopefully did that correctly. If not I'll circle back on Sunday.

@andreaskoepf
Copy link
Contributor

andreaskoepf commented Feb 1, 2025

OK, now formatting & unit tests pass. Since the result in not unique we need a score_answer() function which check whether the right letter-set was used and that the submitted answer is a palindrome.

@Miserlou
Copy link
Collaborator

Miserlou commented Feb 1, 2025

max_length: int = 10 seems like a low limit? Would love to see this done with 32K tokens, for instance..

@andreaskoepf
Copy link
Contributor

@Miserlou I suggest we keep the default config very simple .. in ~1 week we'll have some experience of "curriculum engineering" and can then give some guidelines for "calibrating" the default/baseline complexity which can then be adjusted in a similar way for all tasks scalar value.

@joenorton
Copy link
Contributor Author

I'm fine with changing the default config to whatever is reasonable.

add palindrome score_answer & test
Copy link
Contributor

@andreaskoepf andreaskoepf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect! :-D I like the 0.0 - 0.05 reward leader .. keeps a tangible delta between right and wrong ;-)

        # Check if the answer is a palindrome
        if answer != answer[::-1]:
            return 0.02

        # Check if answer contains the same letters as provided (ignoring order)
        if sorted(answer) != sorted(expected_letters):
            return 0.05

@andreaskoepf andreaskoepf merged commit 00961bc into open-thought:main Feb 3, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants