PPO for conversation datasets

[Reason-Wang]

It seems that all the examples used alpaca-like datasets, where the actor model are given an instruction and generate a response. However, I want to know if there is a way to do PPO in ShareGPT-like conversational datasets, where the actor is updated based on assistant's response (assume we know the reward of each assistant's response). Should I do some masking to mask all user's content.

[younesbelkada]

I am not sure custom masks are supported for PPO, only full padding right / left are support if I am not mistaken, meaning this would be currently not possible to do with PPOTrainer cc @lvwerra

[lvwerra]

Yes masking is supported since the addition of #424. Note that it is even necessary as training on tokens that don't come from the model distribution can have severe side effects!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.