logprobs to assess the confidence of verdict prediction

[Kefan-pauline]

[x] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question

I would like to use logprobs to assess the confidence of verdict predictions in ragas, the implementation will likely be using callbacks. Before implementing it, I would like to know if anyone has already experimented with this and what were your takes on it?

My hope is to reduce the indeterministic nature of the scores from one run to another, by e.g., reducing hallucinations in verdict predictions (i.e. if the confidence is very low for verdict prediction, then rerun the prediction).

@jjmachan
@shahules786

[jjmachan]

hey @Kefan-pauline that is a very good idea and we would love to work with you and help implement it. We can get on a call and discuss this more if you want

but I think the implementation will be similar to how we have it with https://docs.ragas.io/en/stable/howtos/applications/_cost which could be an inspiration but we can discuss it more

[Vidit-Ostwal]

Hi @jjmachan, is there any kind of development on this idea. I would like to learn and contribute to this interesting feature.