DPO数据格式问题，对于同一个instruction有多个chosen和rejected怎么构造数据？

[veraygood]

Reminder

• I have read the README and searched the existing issues.

System Info

[
{
"instruction": "人类指令（必填）",
"input": "人类输入（选填）",
"chosen": "优质回答（必填）",
"rejected": "劣质回答（必填）"
}
]

Reproduction

--

Expected behavior

这里的chosen和rejected可以为list格式吗？我应该怎么构造数据呢

Others

No response

[hiyouga]

仅支持一个，多个请分成多条数据

[veraygood]

@hiyouga
还想请教一下，我有多个response分别对应质量打分，比如对于按照质量打分score排序 response a>b>c>d，以下哪种方式推荐？
方式一：按照score，取相对概念，比如b相比于a更差，b相比于c更好。
[
{
"instruction": prompt,
"chosen": a,
"rejected": b
},
{
"instruction": prompt,
"chosen": b,
"rejected": c
},...]
方式二：设定阈值，比如a>b>thresh>c>d，那么a、b为chosen，c、d为rejected
[
{
"instruction": prompt,
"chosen": a,
"rejected": c
},
{
"instruction": prompt,
"chosen": a,
"rejected": d
},...
]