How can I evaluate Ollama models such as Llama 3.3 using RAGAS?

[HaniaMo]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

[Vidit-Ostwal]

@dosu Can you solve this ?

[dosubot]

You can successfully use RAGAS with Ollama models by using the following code:

from langchain_community.chat_models import ChatOllama
from langchain_community.embeddings import OllamaEmbeddings

ollama_llm = "mistral"
model = ChatOllama(model=ollama_llm)
embeddings_ollama = OllamaEmbeddings(model="mistral")

This solution was suggested in a similar closed issue and received positive feedback ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[HaniaMo]

I successfully run RAGAS to evaluate an Ollama model. However, I get a timeout error during evaluation and NAN values for metrics, although increasing the timeout for the RunConfig.

Evaluating: 0%| | 0/1 [00:00<?, ?it/s]ERROR:ragas.executor:Exception raised in Job[0]: TimeoutError()
{'faithfulness': nan}

**** This is the function I used to calculate the evaluation metrics
def RAGAS_metrics(user_query, model_response, contexts_list):
# 6.2- Run RAGAS metrics
print("\nRunning RAGAS evaluation metrics...")
# To dict
data = {
"question": [user_query], #user_query is string of size 182
"answer": [model_response], #model response is string of size 3361
"contexts": contexts_list, # is a list of the context: [[size=10]]

    }
 dataset = Dataset.from_dict(data)
 results = evaluate(dataset = dataset, metrics=[faithfulness],llm=llm_factory(), 
                    embeddings=embedding_factory(),run_config=RunConfig(max_workers=8,timeout=1000, log_tenacity=True))
 print("Evaluation Results from ragas:")
 print(results)

[jjmachan]

sadly this is a duplicate of #1170
today we don't support ollama model but will get this fixed in the coming weeks