-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow to generate several samples for each prompt #533
Conversation
f558105
to
5cebac7
Compare
a1f9a8a
to
88c8bc6
Compare
19f8eb5
to
4358032
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still learning the outlines/generate/
side of the code-base. If there are any components you would like test cases for, that would help me learn :)
IMHO we shouldn't reshape at all. Users should expect an array of results with the same length as their input prompts. A shape-agnostic decoder should be a final step so we aren't required to convert to decodable array and back again in multiple places.
4358032
to
536cc7d
Compare
4035150
to
6989e99
Compare
Closes #416
TODO
torch.repeat_interleave
so samples of the same sequence in a batch are contiguous (easier for beam search)Questions
(num_samples * batch_size, num_tokens)
, and reshape inSequenceGenerator
? Reshaping in the generator makes the latter much more complex. We could also not reshape at all, and let the user do it manually, which will simplify chained calls in the future.