Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add custom openai host endpoint #1349

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

AnthonyRonning
Copy link

This is a first start at adding a custom openai-compatible endpoint. I have this working with my LiteLLM proxy.

Screenshot 2025-02-22 at 3 38 11 PM Screenshot 2025-02-22 at 3 37 47 PM

Some open issues I've caught so far:

  1. Custom models do work but the "available models" could use work. One option here is to make this list dynamic from the /models endpoint that I believe most openai-compatible APIs should have.
Screenshot 2025-02-22 at 3 42 58 PM
  1. Consequently, the "browse models" link doesn't do anything

  2. Could also add the ability to pass in the model(s) during configuring it.

  3. Could also add the ability to provide multiple custom openai-compatible models but would increase the scope a lot, I would imagine. Could be done in a follow up if there's enough demand.

I'm open to any opinions here for how to handle these issues or postpone for now.

Closes: #885

@yingjiehe-xyz
Copy link
Collaborator

can we directly use the OPEN AI provider? you can set the host and api key to it

@AnthonyRonning
Copy link
Author

can we directly use the OPEN AI provider? you can set the host and api key to it

Definitely possible and could be a lot more simple. Though some of the open questions about handling the models would remain. Still functional without needing to answer those questions yet, it will just require a few manual steps that might not be known to the user right off the bat.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generic Open AI API compatible provider. (Deepseek, Helicone, LiteLLM etc)
2 participants