Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add model manager service for loading/unloading exllama models #3

Open
noizu opened this issue Jul 25, 2024 · 0 comments
Open

Add model manager service for loading/unloading exllama models #3

noizu opened this issue Jul 25, 2024 · 0 comments

Comments

@noizu
Copy link
Member

noizu commented Jul 25, 2024

to avoid the need to preload all models regardless if conditional logic results in their use or not, additionally add additional meta data for controlling how payloads are sent to model (ExLLama already provides encoders for different common chat completion formats) and ideally include image embedding support when available as well as tool usage via prompt injection or provided mechanisms when available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant