Issues setting up

[lpalomares]

Hi guys,

I'm new to LangTest just ran into it while doing so digging on LLM's and find this tool super interesting, I'm currently trying to run an example I found on the HuggingFace website

from langtest import Harness

harness = Harness(
    task = "text-classification",
    model={"model":'lvwerra/distilbert-imdb', "hub":"huggingface"},
    data={
        "source": "huggingface",
        "data_source": "imdb",
        "split":"test[:100]"
    },
    config={
        "tests": {
            "defaults": {"min_pass_rate": 1.0},
            "accuracy": {"min_micro_f1_score": {"min_score": 0.7}},
            'robustness': {
                'titlecase':{'min_pass_rate': 0.60},
                'strip_all_punctuation':{'min_pass_rate': 0.96},
                'add_typo':{'min_pass_rate': 0.96},
                'add_speech_to_text_typo':{'min_pass_rate': 0.96},
                'adjective_antonym_swap':{'min_pass_rate': 0.96},
                'dyslexia_word_swap':{'min_pass_rate': 0.96}
            },
        }
    }
)

harness.generate()
harness.testcases()

But I haven't been able to do so, I keep getting the following errors:

  File "C:\Users\Luis\PycharmProjects\LangTestProject\.venv\Lib\site-packages\typing_extensions.py", line 1174, in <module>
    class TypeVar(typing.TypeVar, _DefaultMixin, _root=True):
TypeError: type 'typing.TypeVar' is not an acceptable base type

These are the versions I'm trying to run:

(.venv) PS C:\Users\Luis\PycharmProjects\LangTestProject> pip show langtest         
Name: langtest
Version: 1.10.0

(.venv) PS C:\Users\Luis\PycharmProjects\LangTestProject> pip show typing-extensions
Name: typing_extensions
Version: 4.5.0

Also I'm trying to do this on PyCharm (Not sure if that may have anything to do with my problem)

Thanks in advanced for any suggestion or help!

[chakravarthik27]

Hi @lpalomares

I hope you're doing well. Can you check the Pydantic and transformers version? Please try this.

pip install langtest[transformers] pydantic==1.10.6

If you encountered the same error, kindly provide the complete error log for us to better understand the issue.

[lpalomares]

Hi @chakravarthik27 !

First of all thank you very much for your reply.

I ran commands pip install langtest[transformers] pydantic==1.10.6 which got me some errors with rust and cargo that I managed to solve but now I'm seeing the following:

(.venv) PS C:\Users\Luis\PycharmProjects\LangTestProject> pip install langtest[transformers] pydantic==1.10.6
Requirement already satisfied: pydantic==1.10.6 in c:\users\luis\pycharmprojects\langtestproject\.venv\lib\site-packages (1.10.6)
Requirement already satisfied: langtest[transformers] in c:\users\luis\pycharmprojects\langtestproject\.venv\lib\site-packages (0.0.1)
Requirement already satisfied: typing-extensions>=4.2.0 in c:\users\luis\pycharmprojects\langtestproject\.venv\lib\site-packages (from pydantic==1.10.6) (4.5.0)
WARNING: langtest 0.0.1 does not provide the extra 'transformers'

This now is causing "Unresolved reference 'langtest'" and "Unresolved reference 'Harness'" so I'm still not able to run this:

from langtest import Harness

harness = Harness(
    task = "text-classification",
    model={"model":'lvwerra/distilbert-imdb', "hub":"huggingface"},
    data={
        "source": "huggingface",
        "data_source": "imdb",
        "split":"test[:100]"
    },
    config={
        "tests": {
            "defaults": {"min_pass_rate": 1.0},
            "accuracy": {"min_micro_f1_score": {"min_score": 0.7}},
            'robustness': {
                'titlecase':{'min_pass_rate': 0.60},
                'strip_all_punctuation':{'min_pass_rate': 0.96},
                'add_typo':{'min_pass_rate': 0.96},
                'add_speech_to_text_typo':{'min_pass_rate': 0.96},
                'adjective_antonym_swap':{'min_pass_rate': 0.96},
                'dyslexia_word_swap':{'min_pass_rate': 0.96}
            },
        }
    }
)

harness.generate()

harness.testcases()

This is the article I was trying to follow: https://huggingface.co/blog/alytarik/langtest-hf-integration

[chakravarthik27]

Hi @lpalomares

Can you try running it on a new virtual environment using pip install langtest[transformers]? Also, please provide your OS information and Python version.

[lpalomares]

Hi @chakravarthik27 !

Like you said I started a new project. This is my current python version:

(.venv) PS C:\Users\Luis\PycharmProjects\TestLangLibrary> python --version                  
Python 3.12.0

And this is the output I got after trying to run pip install langtest[transformers] again:

  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
INFO: pip is still looking at multiple versions of torch to determine which version is compatible with other requirements. This could take a while.
Collecting langtest[transformers]
  Obtaining dependency information for langtest[transformers] from https://files.pythonhosted.org/packages/29/cc/95b5f96013d5fca718702dbf50000d3d8a7719470a670fcf0e4419ce059f/langtest-1.3.0-py3-none-any.whl.metadata
  Downloading langtest-1.3.0-py3-none-any.whl.metadata (13 kB)
  Obtaining dependency information for langtest[transformers] from https://files.pythonhosted.org/packages/20/b5/fb99bb03528a6aa4641bf3518dbea048e6ebbec08c7406b014de400c2372/langtest-1.2.0-py3-none-any.whl.metadata
  Downloading langtest-1.2.0-py3-none-any.whl.metadata (13 kB)
  Obtaining dependency information for langtest[transformers] from https://files.pythonhosted.org/packages/bf/9c/ef9353699828e79d115ece514944ac9bf890ceeba27581bdbbe78052781d/langtest-1.1.0-py3-none-any.whl.metadata
  Downloading langtest-1.1.0-py3-none-any.whl.metadata (7.2 kB)
  Obtaining dependency information for langtest[transformers] from https://files.pythonhosted.org/packages/0a/07/a61bcaabffeb7c51030843ba2fba754858709f24a58736712587366f8e21/langtest-0.0.1-py3-none-any.whl.metadata
  Downloading langtest-0.0.1-py3-none-any.whl.metadata (756 bytes)
WARNING: langtest 0.0.1 does not provide the extra 'transformers'
Downloading langtest-0.0.1-py3-none-any.whl (1.2 kB)
Installing collected packages: langtest
Successfully installed langtest-0.0.1

[notice] A new release of pip is available: 23.2.1 -> 23.3.2
[notice] To update, run: python.exe -m pip install --upgrade pip

I can see the "Collecting langtest[transformers]" at the beggining but eventually I get the "WARNING: langtest 0.0.1 does not provide the extra 'transformers'" message, should I change my Python version?

Thanks!

[chakravarthik27]

Yes, I try with python 3.8 version.

[lpalomares]

Thanks a lot @chakravarthik27!!

Changing to Python version 3.8 solved most of the issues, aditionally I had to install pip install ipython to make it work but that was about it.

I do have one more inquiry related to some of the libraries functionalities:

At first I was using the colab version hosted here: https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/test-specific-notebooks/Accuracy_Demo.ipynb

And wanted to get similar results on my local environment with the following example:

from langtest import Harness

harness = Harness(
    task = "text-classification",
    model={"model":'lvwerra/distilbert-imdb', "hub":"huggingface"},
    data={
        "source": "huggingface",
        "data_source": "imdb",
        "split":"test[:100]"
    },
    config={
        "tests": {
            "defaults": {"min_pass_rate": 1.0},
            "accuracy": {"min_micro_f1_score": {"min_score": 0.7}},
            'robustness': {
                'titlecase':{'min_pass_rate': 0.60},
                'strip_all_punctuation':{'min_pass_rate': 0.96},
                'add_typo':{'min_pass_rate': 0.96},
                'add_speech_to_text_typo':{'min_pass_rate': 0.96},
                'adjective_antonym_swap':{'min_pass_rate': 0.96},
                'dyslexia_word_swap':{'min_pass_rate': 0.96}
            },
        }
    }
)

harness.generate()

harness.testcases()

harness.run()

harness.generated_results()

harness.report()

The questions would be:

1. Is there a way of checking which test cases are being generated with the combination of harness.generate() and harness.testcases()?
2. How could I see the outputs on my local environment that harness.generated_results() and harness.report()?

In the colab example those are showcased like this:

Again thank you very much for your helpfuls responses!