Azure LLM evaluation - Deployment Error

[meenusel]

Describe the bug
Trying to use an Azure API Key to run a LLM evaluation using UpTrain. I received a 404 error message saying that the deployment is not found. However, there is no deployment name parameter when creating the settings for the Eval LLM.

To Reproduce
I followed the steps in this documentation: https://docs.uptrain.ai/llms/azure
Error message:
ERROR | uptrain.operators.language.llm:async_process_payload:103 - Error when sending request to LLM API: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

[ryancole0]

I don't think this is a bug, just looks like a documentation update would be helpful. The Azure OpenAI term "deployment" usually translates to "model" in other frameworks. The following works for me:

settings = Settings(model = 'azure/{}'.format(deployment_name), 
                    azure_api_key=os.getenv('API-KEY'), 
                    azure_api_version=api_version, 
                    azure_api_base=endpoint_url)
eval_llm = EvalLLM(settings)

[terry07]

Hello there.

That library seems great, but I still face issues when I try to sync with my Azure credentials.

None of the relevant comments did help me.

Can you please check for any inconsistency?

{
"name": "ValueError",
"message": "OpenAI API Key is invalid",
"stack": "---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[5], line 5
1 settings = Settings(model = 'azure/{}'.format('gpt-4'),
2 azure_api_key=os.getenv('AZURE_OPENAI_API_KEY'),
3 azure_api_version=os.getenv('OPENAI_API_VERSION'),
4 azure_api_base=os.getenv('AZURE_ENDPOINT'))
----> 5 eval_llm = EvalLLM(settings)

File ~/miniconda3/envs/eval/lib/python3.9/site-packages/uptrain/framework/evalllm.py:101, in EvalLLM.init(self, settings, openai_api_key)
99 response = check_openai_api_key(self.settings.openai_api_key)
100 if not response:
--> 101 raise ValueError("OpenAI API Key is invalid")
103 self.executor = APIClientWithoutAuth(self.settings)

ValueError: OpenAI API Key is invalid"
}

[cthompson-insight]

This is failing when you check for an OpenAI key. The check_openai_api_key only checks if the key is valid using openai.

def check_openai_api_key(api_key):
    
    import openai
    client = openai.OpenAI(api_key=api_key)
    try:
        client.models.list()
    except openai.AuthenticationError:
        return False
    else:
        return True

Needs to also use the AzureOpenAI class to do the check.

[terry07]

Great, I replaced that part with the corresponding part regarding AzureOpenAI check.

from openai import AzureOpenAI

client = AzureOpenAI(
        api_key = openai_api_key,
        api_version=openai_api_version,
        azure_endpoint=openai_azure_endpoint,
            )

[deekshagarg39]

@terry07 Could you please help in How you are passing client to EvalLLM class

[NiklasElsaesser]

@terry07 Could you please help in How you are passing client to EvalLLM class

I would also appreciate to know how to execute this code properly!

[HelloJowet]

I'm using the following workaround until this issue is fixed:

from uptrain import EvalLLM, Settings
from uptrain.framework.remote import APIClientWithoutAuth

class EvalLLMWorkaround(EvalLLM):
	def __init__(self, settings: Settings = None, openai_api_key: str = None) -> None:
		if (openai_api_key is None) and (settings is None):
			raise Exception('Please provide OpenAI API Key')

		if settings is None:
			self.settings = Settings(openai_api_key=openai_api_key)
		else:
			self.settings = settings

		self.executor = APIClientWithoutAuth(self.settings)

OPENAI_API_KEY = "sk-********************"  # Insert your OpenAI key here
eval_llm = EvalLLMWorkaround(openai_api_key=OPENAI_API_KEY)

or

from uptrain import EvalLLM, Settings
from uptrain.framework.remote import APIClientWithoutAuth

class EvalLLMWorkaround(EvalLLM):
	def __init__(self, settings: Settings) -> None:
		self.settings = settings
		self.executor = APIClientWithoutAuth(self.settings)

# ...

[MouliChintakunta-AI]

azure_endpoint= os.getenv("AZURE_OPENAI_API_BASE")
azure_api_version = os.getenv("OPENAI_API_VERSION")
azure_api_key = os.getenv("AZURE_OPENAI_API_KEY")

from uptrain import EvalLLM, Evals, CritiqueTone, Settings
settings = Settings(model ='azure/{}'.format('AIF_V2'), azure_api_key=azure_api_key, azure_api_version=azure_api_version, azure_api_base=azure_endpoint)
eval_llm = EvalLLM(settings)

above code works for me for AzureLLM integration with uptrain