RSO (Statistical Rejection Sampling Improves Preference Optimization)

[gaetanlop]

Implementation of Statistical Rejection Sampling Improves Preference Optimization

Respond to #816.

1. To do:

• Add a function to make generation from the SFT model
• Add a function to score the generated sentences using the reward model
• Example script on how to perform statistical rejection sampling
• Implement the two ranking methods introduced in the paper: ie tournament ranking and first-round ranking

2. To be discussed:

• I have added a generate and score function to trainer.utils as those two functions will also be useful for raft and rest implementations ([WIP] Reward ranked finetuning (RAFT) and Reinforced Self-Training (ReST) #704). It would also be possible to add those functions to a utils.py script inside the examples folder. What do you think?
• Same for the conduct_rejection_sampling function.
• RSO nearly left unchanged the loss function of DPO and mainly introduces a method to generate preference pairs from the target optimal policy. Thus, the training script is the same as dpo.py. I don't think adding another similar script is useful. What's your opinion?
• Not sure of the location to put those RSO example scripts.

cc @kashif @younesbelkada @lvwerra @philschmid

[kashif]

• thanks @gaetanlop I added the `"hinge" loss function which is what the RSO authors also use right?
• I am also refactoring the DPO code a bit to remove the reference model log probs calculation to be done externally to the trainers (as a dataset mapper)
• I had some questions about the seq-2-seq models for these methods (asked in the seq-2-seq merged PR)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[gaetanlop]

Hello @kashif, thanks for looking at this. It looks like the RSO authors experimented both with sigmoid (used in dpo) and hinge (used in SliC) loss function. Can you send the link where you posted the questions for the seq2seq models please? I do not find it

[kashif]

@gaetanlop see here: https://github.com/huggingface/trl/pull/586/files#diff-f2d3c6530916998b7d251573afa193b5f0dfeca5b1a96147fe368c967ea8e079R380

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[lvwerra]

Stay calm stale-bot :D

[hendrydong]

Looks cool!

Are there any recent updates?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.