Conversational dataset support for DPOTrainer

[qgallouedec]

What does this PR do?

Part of #2071

It includes:

• Overwriting the default learning rate of TrainingArguments, so that the user can just use it with the default values. (same as [KTO] learning rate recomentations for kto #2070)
• Extending extract_prompt to support standard dataset.
• Add some doc about vision datasets
• Add script for RLAIF-V dataset generation

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

str1 = "I am Quentin"
str2 = "I am in Lyon"
# What we want:
prompt = "I am"
# What we don't want:
prompt = "I am "

That's why, when the prompt ends with a space, we take idx-1 instead.

[lewtun]

Really nice clean up of the DPO preprocessing and docs 🔥. Great stuff @qgallouedec !

I know we don't currently have any proper regression tests, but WDYT about running the dpo.py script on main and this branch to sanity check the margins / loss look ok before merging?

[lewtun]

WDYT about using trl chat here as a nice demo to show users it exists?

[qgallouedec]

Something like this?

[qgallouedec]

accelerate launch examples/scripts/dpo.py \
    --dataset_name trl-lib/ultrafeedback_binarized \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --learning_rate 5.0e-7 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --gradient_checkpointing \
    --logging_steps 25 \
    --eval_strategy steps \
    --eval_steps 50 \
    --output_dir Qwen2-0.5B-DPO-main \
    --no_remove_unused_columns

https://wandb.ai/huggingface/huggingface/runs/b499t4i2
https://wandb.ai/huggingface/huggingface/runs/jqdy51ld