Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expectation error #1807

Open
amin-kh96 opened this issue Jan 4, 2025 · 10 comments
Open

expectation error #1807

amin-kh96 opened this issue Jan 4, 2025 · 10 comments
Labels
answered 🤖 The question has been answered. Will be closed automatically if no new comments bug Something isn't working question Further information is requested

Comments

@amin-kh96
Copy link

amin-kh96 commented Jan 4, 2025

I am working with ragas and I Have my own embeddings and textual data. i see this error what should I do?

Exception has occurred: ValueError
Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>
  File "C:\Users\Amin\git\ragas-prototype\src\rag_eva.py", line 320, in <module>
    evaluation_report = evaluate(new_ragas_dataset, metrics=metrics, embeddings=ragas_embeddings, llm=custom_llm, column_map=column_map)  # Pass your custom LLM here
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Dataset feature "contexts" should be of type Sequence[string], got <class 'datasets.features.features.Sequence'>
@amin-kh96 amin-kh96 added the question Further information is requested label Jan 4, 2025
@dosubot dosubot bot added the bug Something isn't working label Jan 4, 2025
@hundredeuk2
Copy link
Contributor

hundredeuk2 commented Jan 5, 2025

Based on the error codetype, it sounds like your embedding model is not the problem, but rather the context row in the new_ragas_dataset that makes up the textual data is not a List[str], so you should check the data type first.

When you check the data once, if you load new_ragas_dataset['context'],
Expected output : List[str]

@jjmachan
Copy link
Member

jjmachan commented Jan 7, 2025

thanks for helping out @hundredeuk2 🙂 ❤️

@amin-kh96 like Heondeuk suggest this is likely do to an inconsistency with how you have formatted the data, do check it out.

@jjmachan jjmachan added the answered 🤖 The question has been answered. Will be closed automatically if no new comments label Jan 7, 2025
@amin-kh96
Copy link
Author

@dosubot

in my dataset, the context should be a list of lists like this.
I am using ragas version 0.1.7
{
"contexts": [["context1_chunk1", "context1_chunk2"], ["context2_chunk1"]],
"questions": ["question1", "question2"],
"answers": ["answer1", "answer2"]
}

Copy link

dosubot bot commented Jan 8, 2025

I found similar threads that might help you resolve the issue. In one closed issue, users encountered a similar ValueError and resolved it by ensuring that the "contexts" column in their dataset is a list of strings rather than a list of lists. You can achieve this by flattening the list of lists into a single list of strings. Here's a code snippet that might help:

# Assuming new_ragas_dataset is your dataset
new_ragas_dataset['contexts'] = [item for sublist in new_ragas_dataset['contexts'] for item in sublist]

This transformation ensures that each entry in the "contexts" field is a single list of strings, which should resolve the error you're encountering [1].

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot dosubot bot mentioned this issue Jan 13, 2025
@amin-kh96
Copy link
Author

@dosu
i did it but i face this error
Exception has occurred: TypeError
'Dataset' object does not support item assignment
File "C:\Users\Amin\git\ragas-prototype\src\rag_eva.py", line 178, in
new_ragas_dataset['contexts'] = [item for sublist in new_ragas_dataset['contexts'] for item in sublist]
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: 'Dataset' object does not support item assignment

Copy link

dosubot bot commented Jan 14, 2025

The error you're encountering is due to the fact that the Dataset object in the ragas framework does not support direct item assignment. To modify a column, you can use the map function to transform the data. Here's how you can flatten the "contexts" column:

# Assuming new_ragas_dataset is your dataset
def flatten_contexts(example):
    example['contexts'] = [item for sublist in example['contexts'] for item in sublist]
    return example

new_ragas_dataset = new_ragas_dataset.map(flatten_contexts)

This approach uses the map method to apply the transformation function to each example in the dataset, effectively flattening the "contexts" column.

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@amin-kh96
Copy link
Author

@dosu
I see this what should I do
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.

Copy link

dosubot bot commented Jan 14, 2025

The "Failed to parse output" message indicates that the Ragas framework is encountering issues when trying to parse the output generated by the language model. This could be due to the output not conforming to the expected format or schema.

Here are some steps you can take to troubleshoot and resolve this issue:

  1. Check Output Format: Ensure that the output from your language model matches the expected format. The Ragas framework expects outputs to be in a specific JSON format that can be parsed by the RagasOutputParser. Verify that the output adheres to this format.

  2. Enable Debugging: Set the RAGAS_DEBUG environment variable to True to enable debugging mode. This will provide more detailed error messages and stack traces, which can help identify the root cause of the parsing failure [1].

  3. Review Retry Logic: The framework includes a retry mechanism for parsing outputs. If the initial parsing fails, it attempts to fix the output format and retry. Ensure that this retry logic is functioning correctly and that the output can be corrected within the allowed retries [2].

  4. Inspect Logs: Check the logs for any additional error messages or warnings that might provide more context about the parsing failure. The logs can give insights into what part of the output is causing the issue [1].

By following these steps, you should be able to identify and address the cause of the output parsing errors.

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@amin-kh96
Copy link
Author

@dosu

i see this
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.
Failed to parse output. Returning None.

Copy link

dosubot bot commented Jan 14, 2025

To resolve the "Failed to parse output. Returning None." error in the Ragas framework, you should handle the RagasOutputParserException that is raised when there is an error parsing the output. Here are some steps to help you troubleshoot and fix the issue:

  1. Check Output Format: Ensure that the output from your language model is in the expected JSON format. The expected format is typically:

    {
        "text": "str"
    }

    Make sure the output adheres to this structure [1].

  2. Enable Debugging: Set the RAGAS_DEBUG environment variable to True to enable detailed logging. This can help you identify the specific part of the output that is causing the parsing failure:

    import os
    os.environ["RAGAS_DEBUG"] = "True"
  3. Review Retry Logic: The framework includes a retry mechanism for parsing outputs. If the initial parsing fails, it attempts to fix the output format and retry. Ensure that this retry logic is functioning correctly and that the output can be corrected within the allowed retries [2].

  4. Inspect Logs: Check the logs for any additional error messages or warnings that might provide more context about the parsing failure. The logs can give insights into what part of the output is causing the issue.

By following these steps, you should be able to identify and address the cause of the output parsing errors.

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered 🤖 The question has been answered. Will be closed automatically if no new comments bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants